Qualitative or quantitative characterization of a coating surface

ABSTRACT

The invention relates to a method for providing a coating composition-related prediction program, the method comprising:providing a database (204, 904) comprising associations of qualitative and/or quantitative characterizations of coating surfaces and one or more parameters;training a machine learning model for providing a predictive model (M2, M3) having learned to correlate qualitative and/or quantitative characterizations of one or more coating surfaces with one or more of the parameters; andproviding a composition-quality-prediction program configured for using the predictive model (M2) for predicting the properties of a coating surface to be produced from one or more input parameters; and/orproviding a composition-specification-prediction program configured for using the predictive model (M3) for predicting, based on an input specifying at least a desired coating surface characterization, one or more output parameters related to a coating composition predicted to generate a coating surface having the input surface characterizations.

TECHNICAL FIELD

The invention relates to the identification of coating defects and tothe characterization of coating surfaces, in particular coating surfacesbased on coating compositions for paints, varnishes, printing inks,grinding resins, pigment concentrates or other coating compositions.

BACKGROUND

Coating compositions are complex mixtures of raw materials. Commoncompositions or recipes or formulations for coating compositions containabout 20 raw materials or more. This distinguishes them from alloys,which are admixtures of a limited set of metals, or a metal combinedwith one or more other elements. While an alloy is a solid solution ofmetal elements which typically has a single phase and are defined by ametallic bonding character, coating compositions typically are complexmixtures of raw materials. Common compositions or formulations forpaints, coatings, printing inks, grinding resins, pigment concentratesor other coating materials contain about 20 or more raw materials. Thesecompositions consist, for example, of raw materials selected fromsolids, such as pigments and/or fillers, binders, solvents, resins,hardeners and various additives, such as thickeners, dispersants,wetting agents, adhesion promoters, defoamers, surface modifiers,leveling agents, catalytically active additives, such as drying agentsand catalysts, and specially active additives, such as biocides,photoinitiators and corrosion inhibitors. For the dispersion of solids(e.g. pigments, fillers or dyes) in liquid media, dispersants (alsocalled dispersing additives) are usually used to achieve effectivedispersion of the solids, to reduce the mechanical shear forces requiredfor dispersion and, at the same time, to realize the highest possibledegree of filling. The dispersants assist in breaking up agglomerates,act as surface-active materials to wet and/or coat the surface of thesolids or particles to be dispersed, and stabilize them againstundesirable re-agglomeration. Dispersants facilitate the incorporationof solids, such as pigments, dyes and fillers, which, as importantcompositional constituents, essentially determine the optical appearanceand physicochemical properties of such systems. For optimum utilization,these solids must on the one hand be uniformly distributed in theformulations, and on the other hand the distribution once achieved mustbe stabilized. A large number of different substances are used today asdispersants for solids. In addition to very simple, low-molecularcompounds, such as lecithin, fatty acids and their salts and alkylphenolethoxylates, more complex high-molecular structures are also used asdispersants. Here, it is specifically amino- and amido-functionalsystems that are widely used.

For this reason, the combinatorial complexity of coating compositions ishuge, typically much larger than that of alloys or other types ofchemical products. Moreover, the process of applying the coatingcomposition on a surface is highly complex and has a strong impact onthe coating quality. The process of generating, stabilizing and applyinga coating composition is much more complex than the generation ofalloys, because coating compositions may segregate into different phasesduring storage or application.

Due to the large number of coating composition components and thecomplexity of the manufacturing and application process, identifying thecoating composition whose physical or chemical properties are suited fora particular application scenario or substrate material is a highlycomplex and challenging task.

Paint and varnish coatings can have a variety of defects that negativelyinfluence the appearance or technical properties of the coated object.The coating defects can be, for example, foam, craters, clouding,levelling problems, corrosion, wetting problems, floating of pigments(floating), sagging, agglomeration, or bubble formation, whereby severaldefects can occur simultaneously and can influence each other. Toinvestigate and avoid these problems, test substrates are coated withthe formulation during formulation development and examined for defects.Depending on the intended field of application, different substrates areused, for example wood, plastic, paper/cardboard, glass, or metal.Furthermore, different pretreatments of the substrate are possible, andthe pretreatments may further complicate matters. Due to the number ofinterdependent process parameters, large number of coating compositions,pretreatment approaches and substrate types, it is currently impossibleto predict whether or not a particular coating composition will providea coating of acceptable quality if applied on a particular substrate.Therefore, the coating surface quality can currently only be determinedretrospectively.

Currently, the defects are assessed visually by a human being, e.g. anemployee. This purely visual assessment is typically verycoarse-grained, highly subjective and hardly reproducible. As aconsequence, the identification of defects and the assessment of thequality of the coating surface may require a great deal of experience onthe part of the employee but may nevertheless vary strongly from personto person which makes it difficult to compare the results. In addition,the manual evaluation of coating surfaces is time-consuming and henceexpensive.

SUMMARY

The invention aims at providing an improved method and correspondingsystem for the characterization of surface defects in a coating surfaceand to the use of the resulting programs and information in the contextof the manufacturing of coating compositions as specified in theindependent claims. Embodiments are given in the dependent claims.Embodiments of the present invention may be freely combined with eachother, provided they are not mutually exclusive.

In one aspect, the invention relates to a method for qualitative and/orquantitative characterization of a coating surface. The methodcomprises:

-   -   processing a digital image of the coating surface by a        defect-identification program, the defect-identification program        being configured to recognize types of coating surface defects,        e.g. by recognizing patterns, each pattern representing one of        the coating surface defect types; and    -   outputting a characterization of the coating surface, the        characterization being computed as a function of coating surface        defects recognized by the defect-identification program.

Embodiments of the invention may have the advantage that acharacterization of coating surface is provided by means of adefect-identification program and hence in a reproducible, objective andfast manner.

For example, the output characterization may indicate that the coatingsurface is free of a defect type D1 but may comprise defects of defecttype D2 and D3. The output may also indicate that the coating surface isfree of any coating defect. According to other examples, the output maybe more concrete and in addition indicate the amount or extent of thecoating defect, e.g. by indicating that the surface comprises a minordefect of type D2 and a severe defect of type D3. In other examples, thecharacterization can comprise a numerical characterization of the extentof the defects and/or an indication of individual pixels in the digitalimage showing a particular type of defect or belonging to a particularinstance of a particular defect type.

Embodiments of the invention may have the advantage that the occurrenceand/or locations of defects in a coating surface are detected fullyautomatically and used for automatically computing a characterization ofthe coating surface in dependence on the type and/or extent of the oneor more defects identified in the image analysis procedure. Hence, alarge number of digital images can be evaluated and annotated with thecomputed characterization of the respectively depicted coating surfacefully automatically. This may be particularly useful in the context of ahigh throughput facility for testing and/or manufacturing coatingcompositions. The automated determination of the coating surfacecharacterization increases the transparency and reproducibility of theassessment of the quality and other properties of the coating surface.

Embodiments of the invention may have the further advantage that theautomatically identified coating surface defects and thecharacterization of the coating surface derived therefrom can be used asdatabases for performing many different forms of data analysis. Inparticular, the computed characterizations can be used as a qualitativeand/or quantitative indicator of the quality of a coating surface and ofthe quality of the coating composition, the quality or suitability ofthe coating composition manufacturing process and/or the quality orsuitability of the coating application process (including apre-treatment process of the surface, if any, the substrate type and/orthe application equipment) used for manufacturing the coatingcomposition and the coating surface.

The automated computation of qualitative and/or quantitative coatingsurface characterizations may enable an automated analysis of largeamounts of data and may ensure comparability of the surface qualitycharacterizations associated with different coating compositions,coating manufacturing process parameters, and/or coating applicationprocess parameters. This is particularly useful in the context ofproducing and testing many different varieties of a coating compositionto identify the optimum coating composition for a substrate or use casescenario.

Embodiments of the invention may be advantageous in particular in thecontext of the production of paints, varnishes, printing inks, grindingresins, pigment concentrates and other coating materials, since areproducible, objective characterization of surface properties ofrespective coating surfaces was previously not available. A predictionof the coating surface quality was not possible due to the subjectivityof manual descriptions of the coating surfaces and due to the largenumber of components and their interactions.

For example, foam is a phenomenon which frequently occurs during themanufacture and application of coatings and printing inks. The cause offoam is the introduction of gas into the liquid material. This can occurby:

-   -   mechanical introduction of air during manufacture by stirring        and mixing    -   displacement of air during wetting of pigments and fillers    -   mechanical introduction of air during applications such as        rolling, spraying, printing    -   displacement of air when coating porous substrates.

Dried foam leaves surface defects (e.g. bubble defects) in the paintfilm. The foam not only affects the visual appearance, it also reducesthe protective function of the coating. Virtually all the components inthe coating formulation can influence foaming behavior positively ornegatively. The substrate and method of application also have an impacton foaming behavior. Therefore, a defoamer typically is a necessarycomponent in the formulation of a coating composition.

Applicant has observed that the effectiveness of the defoamer of acoating composition depends on its partial incompatibility in thecoating medium. This permits formation of defoamer droplets in thesystem, without causing film or surface defects due to excessiveincompatibility. Thus, a key feature of all defoamers is their targetedand controlled incompatibility with the medium that is to be defoamed. Adefoamer that is too compatible does not migrate into the foam lamellaspecifically, it is present in the entire coating film. The defoamingeffect is then either low or non-existent. Too much incompatibilitycauses problematic coating defects such as turbidity or cratering.Choosing the correct defoamer is therefore a kind of “balancing act”between compatibility and incompatibility. Defoaming is thus always acompromise between efficiency and compatibility.

As a result of the multitude of different coating systems, there is no“one” defoamer that is optimally suited to all formulations. To ensurethat a suitable product can be provided for any purpose, a range ofdefoamers is required. The defoaming effect can be finely adjusted byvarying the dosage: In general, better defoaming is achieved the moredefoamer is used. However, this may also increase the defects (e.g.cratering) or rather these may become more visible. Cratering defectsare defects resulting from the incompatibility of components which maybe caused e.g. by too much defoamer and/or the wrong defoamer. Reducingthe dosage of the defoamer prevents film defects, though the defoamingeffect may, in some circumstances, not be sufficient. Furthermore,applicant has observed that in some coating compositions, increasing theamount of the defoamer(s) above a certain amount may actually decreasethe defoaming effect. Hence, identifying one or more defoamers whichprovide a coating surface with no or minimal defects is a highly complextask.

According to embodiments of the invention, the coating composition usedfor generating the coating surface comprises a combination of two ormore different defoamers whose type and amount was chosen such that acompromise between defoaming efficiency and compatibility/mixabilitywith other components of the composition is achieved.

Hence, the selection of the type and amount of the one or more defoamersin a coating composition is a highly complex task. The situation isfurther complicated by the fact that the type and amount of the selecteddefoamer(s) interrelates with the type and amounts of other componentsof the coating composition, with the coating manufacturing processparameters and/or the coating application process parameters. Minorchanges in one of these parameters may have a strong effect on the typeand extent of surface defects observed in a respective coating surface.A manual characterization of the quality of a coating surface has so farbeen a major obstacle in the identification of changes in respect to thecoating composition and associated process parameters that provide foran improvement in the coating surface qualities.

Applicant has observed that the automated detection of coating defectsof particular defect types such as bubble defects and cratering defectsmay allow training a further machine learning model (e.g. the models M2,M3 described below for some embodiments of the invention) on datacomprising various coating composition specifications associated withrespectively observed coating defects in order to automatically computeand output a specification of an improved version of a coatingcomposition (M3) or to compute and output coating surfacecharacterizations, e.g. coating defects, that are expected for a given(complete or incomplete) coating composition (M2): for example, themachine learning model may learn during a training phase that bubbles onthe coating surface are indicative of ineffectiveness of a defoamer andthat cratering defects in the coating surface are an indication ofincompatibility of the defoamer and the coating medium. In the firstcase, the model may suggest replacing the currently used defoamer by adefoamer being more effective. In the latter case, the model may suggestreplacing and/or supplementing the currently used defoamer by/with adefoamer predicted to be more compatible with the coating medium. Incase the coating composition already comprises an “effective” defoamerDF1 and a “compatible” defoamer DF2 in a particular ratio r=DF1:DF2, themodel in the first case may suggest increasing the ratio and in thelatter case suggest decreasing the ratio.

Hence, according to embodiments of the invention, thecomposition-specification prediction model is used for predicting aratio of at least two different defoamers, in particular the ratio of afirst defoamer with high defoaming effectiveness and low coating mediumcompatibility and a second defoamer with low defoaming effectiveness andhigh coating medium compatibility, whereby the ratio is predicted tominimize both the occurrence of bubble defects and cratering defects.

Other defects may be caused by another component and/or manufactureprocess parameters.

According to some embodiments, the characterizations of the coatingsurfaces output by the defect-identification program comprisesfine-granular quantitative characterizations, e.g. a numerical valuewithin a continuous scale or within a set of at least 10 differentpredefined numerical values or value ranges. The defect-identificationprogram can be configured to transform the fine-granular quantitativecharacterizations into coarse-granular quantitative characterizations tomake the automatically computed characterizations comparable to anexisting, coarse-grained data set. For example, the existing,coarse-grained data set may have been created manually. Thecoarse-granular quantitative characterization can be a numerical valuewithin a set of less than 10 different predefined numerical values orvalue ranges.

The automated transformation may increase the data basis for variousdata analysis or machine learning purposes by proving a mix of manuallylabeled and automatically labeled digital images of coating surfaceswhich are comparable to each other.

According to other embodiments, the output of the defect identificationprogram can be coarse-grained from the beginning. For example, theoutput can be a mere image classification output comprising image classlabels such as “defect-free coating surface”, “coating surface with(minor) bubble defects” or “coating surface with (minor) bubble defectsand (severe) cratering defects.

According to embodiments, the method comprises calculating, by thedefect-identification program, a measure for the recognized defects forcomputing the qualitative and/or quantitative characterization of thecoating surface; and outputting the qualitative and/or quantitativecharacterization of the coating surface.

For example, a coating surface may comprise two or more defects of oneor more different defect types and of varying extents. The calculatedqualitative and/or quantitative measures of each of the defects can beused, e.g. aggregated, for obtaining a qualitative and/or quantitativecharacterization of the coating surface which integrates theautomatically obtained measures of the automatically identified defectsin the coating surface.

The identification of individual defects and the automated determinationof the measures of the individual defects may have the advantage thatthe number, the type, the size and further properties of each individualdefect in a coating surface is objectivized. Embodiments of theinvention are much less subjective than prior art approaches based onmanual/visual assessment of individual defects and/or coating surfacequalities by the staff. The visual evaluation was not quantitative andonly allowed a very rough classification of the results according tocoarse-grained quality classes or a coarse-grained grading system.

According to embodiments, the measure of the defect is or comprises aquantitative measure selected from a group comprising: the area of thedefect (in absolute terms or relative to the size of the coating surfacedepicted in the image), the number of bubbles, depressions or otherdefect-associated patterns observed in the image, the maximum, minimumand/or average size of the bubbles or depressions or other patternsrepresenting a defect observed in the image.

In addition, or alternatively, the measure of the defect is or comprisesa quantitative measure. The quantitative measure can in particular bethe type of the defect identified. For example, the defect type can beselected from a group comprising a cratering defect, an abrasion defect,an adhesion failure defect, an alligatoring defect, a bleeding defect, ablistering defect, a bloom defect, a bridging defect, a bubbling defect,a cathodic disbanding defect, a checking defect, a cissing defect, acobwebbing defect, a cracking defect, a crazing defect, a crowsfootingdefect, a delamination defect, a fading defect, a flaking defect, agrinning defect, a heat defect, an impact defect, an intercoatcontamination defect, a mud cracking defect, an orange peeling defect, apeeling defect, a pinholes defect, a rippled coating defect, a runsdefect, a rust rashing defect, a rust spotting defect, a rust stainingdefect, a sags defect, a settlement defect, a skinning defect, a solventlifting defect, a solvent popping defect, a stress cracking defect, anundercutting defect, a wrinkling defect.

According to embodiments, the method further comprises:

-   -   determining at least one coating surface defect type to be        identified;    -   automatically determining one or more illumination angles and/or        one or more image acquisition angles allowing the acquisition of        a digital image that enables the defect identification program        to compute the characterization of the coating surface depicted        in the image in respect to the at least one determined defect        type;    -   positioning one or more light sources at the determined        illumination angle relative to the coating surface; and/or    -   positioning one or more cameras (preferably one camera) at the        determined one or more image acquisition angles relative to the        coating surface; and    -   after the positioning of the light source(s), the camera(s) and        the coating surface relative to each other, using the camera(s)        for acquiring the digital image(s) of the coating surface.

In addition to or instead of positioning the light sourcesautomatically, the defects-identification-program can be configured togenerate a feedback signal indicating that and/or how adjustment of thepositioning of the one or more light sources and the coating surfacerelative to each other can be adapted such that the illumination anglelies within the identified predefined illumination angle range.

The illumination angle and/or image acquisition angle suitable fordetecting a surface defect may vary depending on the type of defectand/or the type of substrate used.

For example, for the detection of bubble defects, preferably a tiltedillumination angle, e.g. an angle between >0° to 70°, in particularabout 45°, and between 110° to <180°, in particular about 135°, can beused as this provides bubble-induced shadow formations with sufficientcontrast.

The preferred image acquisition angle for detecting bubble defects canbe any one of >0°-360°, preferably 0-180°, more preferred 10°-170°.

To the contrary, cratering defects in coating surfaces can be identifiedand characterized using a basically orthogonal image acquisition angleand an orthogonal (background) illumination angle. For example, for thedetection of cratering defects, preferably an image acquisition anglebetween 70° to 110° can be used. For transparent substrates (e.g. glassor transparent plastic), a preferred illumination angle is between 250°to 290° and the one or more light sources are backlights(transmitted-light illuminator(s)) positioned such that the lightemitted by the light sources passes the transparent substrate and thecoating surface basically orthogonally before it is captured by the oneor more cameras. The basically orthogonal illumination angle reduceslight reflections and allows to determine the thickness of the coatingat the bottom and the sides of the craters and/or the depth of thecraters. For opaque substrates, other illumination angles between >0°and <180° may be used, preferably orthogonal illumination angles, e.g.illumination angles between 70° to 110°.

Embodiments of the invention which control the relative position of theone or more cameras, the one or more light sources and the coatingsurface as well as the illumination angle(s) and image acquisitionangle(s) may have the advantage that the digital images to be providedas input to the defect-identification program are ensured to be takenfrom a perspective and under illumination conditions which arecomparable and which allow to identify and characterize the depicteddefects. In case the defect-identification program was obtained based ona machine learning approach, embodiments of the invention controllingthe relative positions and the illumination angle(s) and imageacquisition angle(s) may ensure that the conditions used for acquiringthe digital image are similar to the conditions used for obtaining thetraining images from which the defect-identification program wasderived. By exerting a tight control of the image acquisition process,embodiments may ensure that the acquired digital images are comparableand can be processed by the defect-identification programreproducibility and accurately.

According to embodiments, the processing of the digital image furthercomprises:

-   -   performing, by the defect-identification program, a        classification of the digital image in respect to the type        and/or amount of surface defects depicted therein and/or a        semantic segmentation of the image based on one or more surface        defect types depicted therein and/or object detection and/or        instance segmentation of the image, thereby automatically        assigning one or more labels to the whole digital image, to        image regions and/or to individual pixels, each label being        indicative of the type of defect identified in the digital        image; and    -   outputting the one or more assigned labels.

According to some embodiments, the defect identification program (whichmay be based on publicly available programs such as Mask R-CNN and mayin addition comprise one or more additional program modules or functionse.g. for generating a GUI, for exchanging data with a network, adatabase and/or a HTE or for performing further predictive tasks), isconfigured to perform image classification in respect to the type and/oramount of surface defects depicted therein and/or to perform a semanticsegmentation of the image based on one or more surface defect typesdepicted therein and/or to perform object detection and/or instancesegmentation of the image.

Different approaches can be used for creating different types of labelsfor different types defects and/or use case scenarios. In some examples,combinations of two or more labels of different types can be assigned toan image, image region or pixel.

For example, image classification is the process of classifying thedigital image in respect to the type(s) of coating defects depictedtherein. A class label may look like “surface free of defects”, “surfacewith bubble defects” or “surface with severe bubble defects”. A classlabel does not identify the position and number of individual defectinstances.

Semantic segmentation is the process of identifying coating defects ofone or more different defect types at the pixel level, whereby noinformation regarding the outline of individual defect instances isobtained. This means that the semantic segmentation provides informationon the type and location of different defect types at the pixel level,but in case several defect instances of the same type overlap, theycannot be recognized as separate defect instances. A coating defectsemantic label may look like “bubble defect(s) identified at pixels {xycoordinates of all pixels in the digital image depicting a bubbledefect}” or “cratering defect(s) identified at pixels {xy coordinates ofall pixels in the digital image depicting a cratering defect}”. Acoating defect semantic label identifies the location of one or moredefect types identified in the image at the pixel level and hence allowsa rough quantification of the extent of the defect. A defect typesemantic label may have the advantage of providing sufficiently detailedinformation for identifying the location and extent of one or moredifferent defect types but may not allow identifying individual defectinstances.

“Object detection” in this context is the process of identifying coatingdefect instances of one or more different defect types and theirapproximate location in the digital image. A coating defect object labelmay look like “bubble defect instance #234 at pixels {xy coordinates ofa (typically rectangular) bounding box comprising this bubble defectinstance #234}” or “bubble defect instance #554 at pixels {xycoordinates of a (typically rectangular) bounding box comprising thisbubble defect instance #554}”. A coating defect object label identifiesindividual instances of one or more defect types identified in the imageand also identifies the rough positions of these defect instances in theimage not at the pixel level, but at the level of bounding boxes. Thecoating defect instance labels can be graphically represented in theform of a digital image in which bounding boxes predicted to comprise anidentified defect type instance are highlighted. Defect type objectlabels may have the advantage of providing a large amount of detailedinformation which allow a quantification of a given defect type andcounting defect type instances. That the positional information is basedon coarse grained bounding boxes may ease the storing and processing ofthose labels.

Defect instance segmentation is the process of identifying coatingdefect instances of one or more different defect types and theirrespective location in the image at the pixel level. A coating defectinstance label may look like “bubble defect instance #234 at pixels {xycoordinates of a set of pixels in a blob depicting this bubble defectinstance #234 }” or “bubble defect instance #554 at pixels {xycoordinates of a set of pixels in a blob depicting this bubble defectinstance #554}”. A coating defect instance label identifies individualinstances of one or more defect types identified in the image and inaddition identifies the pixel positions of these defect instances in theimage. The coating defect instance labels can be graphically representedin the form of a digital image in which image segments representing anidentified defect type instances are highlighted. For example, differentsegments can be graphically represented as segments having a particularcolor which differ from the color of the intact coating surface. In someembodiments, different segments representing different instances of thesame type of defect have different colors. In addition, oralternatively, different segments representing instances of differentdefect types can have different colors. Defect type instance labels mayhave the advantage of providing a large amount of detailed information,but the storing and processing of those label may require moreresources.

Identifying the approximate (bounding boxes) or detailed (pixel-based)location of defect types and/or defect type instances may have theadvantage that the positional information can be further-processedeasily by the defect-identification program, e.g. for computing thefraction of pixels of an image covered by the defect, for computing andhighlighting identified defects using e.g. elliptic, circular,spine-shaped lines, contours or segment edges and the like. On the otherhand, providing a graphical representation of the pattern instances mayhave the advantage that the identified defects can be easily recognizedby a human being. For example, the defect-identification program cangenerate a graphical user interface (GUI) configured to display imagesegments having been identified to represent a coating defect of aparticular type by a respective color or hatching. For example, imagesegments identified to represent bubbles or depressions can behighlighted in a particular color, e.g. yellow. Alternatively, only thesegment borders can be highlighted. Providing a combination ofcoordinate information and a graphical representation may have theadvantage that the output of the defect-identification program caneasily be processed and interpreted both by software and humans.

Applicant has observed that region proposal networks, e.g. the regionproposal network provided by the Mask R-CNN program, can accuratelyidentify and characterize a large variety of coating defect types.However, according to some embodiments, other image analysis methods,and in particular blob detection methods, are used instead of or inaddition to the Mask R-CNN program for identifying coating defectsand/or for characterizing coating surfaces comprising these defects.

For example, the YOLO neural network (J. Redmon and A. Farhadi: “Yolov3:An incremental improvement”, arXiv, 2018) can be used for analyzingdigital images for identifying coating defects.

According to further examples, one or more of the following blobdetection image analysis methods have been observed by the applicant tobe able to accurately identify and/or characterize one or more coatingdefect types, in particular bubble defects and/or cratering defects:

-   -   Simple thresholding: the method comprises setting a hard pixel        color boundary for performing a binary image segmentation into        defects and background. The “color” can be an intensity value of        a monochrome image or an intensity value of a color channel of a        multi-channel image.    -   Otsu thresholding: the method comprises automatically        calculating a global threshold value from an intensity histogram        of a bimodal image.    -   Adaptive or dynamic thresholding: the method comprises        calculating a threshold value separately for small regions of        the image, leading to different threshold values for different        regions.

According to other embodiments, one or more of the following blobdetection methods are used:

-   -   “Find contours”: the method comprises detecting contours, which        are all the points with a similar intensity in a given image.    -   Edge detection: the method comprises using detected edges, which        can be described as discontinuous local features, to separate        background and blobs.    -   Clustering: the method comprises separating an image into        similar pixel segments with clustering techniques such as        k-means.    -   Watershed transformation: the watershed transformation method        comprises treating the image it operates upon like a topographic        map, with the brightness of each point representing its height,        and finds the lines that run along the tops of ridges.

According to embodiments, the digital image is pre-processed in order toincrease the accuracy of the subsequent image processing steps. Forexample, the pre-processing of the image may comprise one or more of:

-   -   Flood fill: the method comprises adapting adjacent values in an        image based on their similarity to an initial seed point in        order to minimize noise and improve blob detection.    -   Morphological transformations: the method comprises performing        the basic morphological operators Erosion and Dilation and        decreasing or increasing the amount of pixels used by features        on the image, such as edges.    -   Smoothing: the method comprises performing a pre-processing of        the image to remove noise via e.g. Gaussian filters.

According to embodiments, the method further comprises installing and/orinstantiating the defects-identification program on a data processingsystem comprising a graphical user interface (GUI). The data processingsystem can comprise or can be operatively coupled to a camera. Thedefect-identification program can be configured to generate a GUI whichis displayed to a user via a screen of the data processing system. Inresponse to a user action via the GUI, the defects-identificationprogram acquires the image of the coating surface via the camera. Thedefects identification program uses the acquired image as the image thatis processed by the defects-identification program for automaticallyidentifying the defect pattern, for computing the measures of the defectpattern and for computing the qualitative and/or quantitativecharacterization of the coating surface. Then, the defectsidentification program performs the outputting of the computedcharacterization via the GUI or another output interface of the dataprocessing system.

According to embodiments, the defects-identification program isoperatively coupled to a camera and is configured for

-   -   determining whether the camera is positioned within a predefined        distance range and/or within a predefined image acquisition        angle range relative to the coating surface; for example, the        distance range and/or angle range can be adapted to enable        acquisition of images from a similar relative position as used        for acquiring training images for generating the predictive        model of the defects-identification program; in addition, or        alternatively, the predefined distance range can be adapted to        enable acquisition of images having at least a predefined        minimum resolution (pixels per depicted surface area); and

-   in dependence on the result of the determination:    -   generating a feedback signal for the user and/or for the camera        whether adjustment of the camera position is required; and/or    -   automatically adjusting the relative position of the camera and        the coating surface; and/or    -   enabling the camera to acquire images selectively in case the        camera is within the predefined distance and/or image        acquisition angle range.

This may have the advantage that the acquisition and analysis of animage of a coating surface is prohibited from the beginning if thecurrent setting of the image acquisition system does not allow theacquisition of digital images under conditions required by thedefect-identification program in order to perform an accurate andreproducible defect identification and coating surface characterizationprocess.

For example, the predefined distance range and/or the predefined imageacquisition angle range can be a range which is explicitly or implicitlygiven before the digital image of the currently presented coatingsurface is acquired.

The resolution of a digital image is indicative of the number of pixelsused in this image for depicting a coating surface area of a given size.For example, the resolution can be given as the number of pixels per cm²of the depicted coating surface, or the number of pixels per inch².

According to some examples, the predefined minimum resolution can beprovided in the form of an absolute resolution value such as at least 5px×5 px per 0.05 mm² of the depicted surface, wherein “px” is a pixel.More preferentially, the predefined minimum resolution is at least 10px×10 px per 0.05 mm² of the depicted surface, or is at least 15 px×15px per 0.05 mm²of the depicted surface.

According to other examples, the predefined minimum resolution isprovided in the form of an absolute resolution value range. The lowerlimit of the range can be, for example, at least 5 px×5 px per 0.05 mm²of the depicted surface, or at least 10 px×10 px per 0.05 mm² of thedepicted surface, or at least 15 px×15 px per 0.05 mm² of the depictedsurface. The upper limit of the range can be, for example, 10.000×10.000px per 0.05 mm² of the depicted surface.

For example, for the defect type “microbubbles”, the minimum resolutioncan be 8 px×8 px per 0.05 mm² of the depicted surface. A “microbubbledefect” as used herein is a bubble defect characterized in that thediameter of the bubbles is below 0.5 mm.

In addition, or alternatively, for the defect type “macrobubbles”, theminimum resolution can be 5 px×5 px per 0.6 mm² of the depicted surface.A “macrobubble defect” as used herein is a bubble defect characterizedin that the diameter of the bubbles is at least 0.5 mm.

In addition, or alternatively, for the defect type “craters”, theminimum resolution can be 8 px×8 px per 0.05 mm² of the depictedsurface.

According to some embodiments, the defects-identification program isconfigured to computationally reduce the resolution of the digital imagein case the resolution exceeds the upper resolution range such that thereduced resolution of the digital image is within the predefinedresolution range. This may have the advantage that the image with thereduced resolution range has a resolution which is identical orsufficiently similar to the resolution of the training images used fortraining the predictive model. This may ensure that thedefects-identification program can accurately perform the predictioneven in case the resolution of the at least one camera should be muchhigher than the resolution of the training images (which was observed tobe an error source).

According to embodiments, the predefined minimum resolution and/or thepredefined resolution ranges are defect type specific. For example, foreach of one or more different defect types, a respective predefinedminimum resolution or resolution range can be stored in association witha defect-type identifier of said defect type in a storage medium.

The applicant has observed that the size of the different coating defecttypes varies significantly. The applicant has further surprisinglyobserved that the use of training images whose resolution is adapted tothe size of the coating defect type to be learned during the trainingmay have a strong impact both on the computational resources requiredduring the training and the accuracy of the generated predictive model:if the resolution is too high, the training will require large amountsof CPU capacity, memory and time and may in some cases lead tosub-optima results. If the resolution is too low, the accuracy of thetrained model will be poor. However, due to the fact that differentdefect types have different sizes and/or may have different degrees offiligree, the minimum resolution and/or the optimum resolution range tobe used both at training time and test time was observed to depend onthe defect type.

According to embodiments, the defects-identification-program dynamicallyreceives a selection of one or more defect types to be identified,dynamically determines, for each of the selected defect types, adistance between the at least one camera and the surface which ensuresthat the image to be acquired will have resolution which is at least theminimum resolution stored in association with an identifier of thisdefect type in a data storage medium. Alternatively, thedefects-identification-program dynamically determines, for each of theselected defect types, a distance between the at least one camera andthe surface which ensures that the image to be acquired will have aresolution which represents the optimum resolution range of this defectstype and which is stored in association with an identifier of thisdefect type in a data storage medium. Then, thedefects-identification-program triggers a relative movement of the atlast one camera and/or the presented surface such that the camera isenabled to acquire the image in a resolution which is at least theminimum defect-type-specific resolution and/or which is within theoptimum resolution range of this defect type. In case multipledefect-types are selected, the camera repeats the adaptation of therelative distance for each of the selected defect types. In addition, oralternatively, the defects-identification-program outputs a feedbacksignal enabling a user to manually or semi-automatically adjust therelative position.

The predefined image acquisition angle range may depend, for example, onthe image acquisition angles of the at least one camera used foracquiring the training images and may be chosen such that the imageacquisition angle of the at least one camera to be used for acquiring animage of the currently presented coating surface is sufficiently similarto the image acquisition angle(s) used for obtaining the trainingimages. The term “similar” may mean e.g., having the same imageacquisition angle +/−<5%, or having the same image acquisition angle+/−<10%, or having the same image acquisition angle +/−<15%, or havingthe same image acquisition angle +/−<20%, or having the same imageacquisition angle +/−<40% as the image acquisition angle used foracquiring the training images.

According to some examples, the predefined distance range is a range ofdistances between the at least one camera (to be used for acquiring thedigital image of the currently presented coating surface) and thepresented coating surface which will result in the acquisition of adigital image having a resolution of at least the minimum resolution,whereby the minimum resolution is a defect-type specific minimumresolution associated with a particular defect, whereby the predictivemodel was trained to recognize this defect type based on training imageshaving at least the said minimum resolution.

Dynamically adapting the camera-surface-distance may be beneficial asthe type and/or the configuration of the at least one camera used foracquiring the digital image of the currently presented coating surfacemay differ from the type and/or configuration of the camera(s) used foracquiring the training images. Automatically determining whether the atleast one camera has a distance to the surface area which allows toacquire a digital image having a resolution which is at least thepredefined minimum resolution or which is within a predefined resolutionrange may allow ensuring that the camera is only enabled to acquire thedigital image if the predictive model will likely be able to identifycoating defects of one or more different types. Likewise, automaticallydetermining whether the at least one camera has an image acquisitionangle which allows to acquire a digital image from a similar angle asused for acquiring the training images may allow ensuring that thecamera is only enabled to acquire the digital image if the acquisitionangle is similar to the acquisition angle used for acquiring thetraining images.

According to embodiments, the predefined image acquisition angle rangesare defect type specific. For example, for each of one or more differentdefect type, a respective predefined image acquisition angle range canbe stored in association with a defect-type identifier in a storagemedium.

According to some embodiments, the training images used for training thepredictive model(s) of the defect-identification program do not onlydepict a coating surface, but also depict a reference object of knownsize. For example, the reference object can be positioned on or next tothe coating surface. The reference object can be, for example, a coin, apiece of paper, or any kind of object with a known size. The same typeof reference object is placed on or next to the coating surfacecurrently presented to the at least one camera. According to someembodiments, at training phase of the predictive model, the number ofpixels used for depicting the reference object in each of the trainingimages is determined for computing a resolution of each training imageas a function of the known reference object size and the determinednumber of reference-object-pixels. The computed resolutions are stored,according to some examples, as part of the training data, e.g. on aper-training-image basis. In some examples, the minimum resolution usedfor obtaining the training images (used for obtaining all trainingimages or used only the images of one or more specific defect types) isused as the lower limit of the predefined resolution range.

According to some embodiments, the defects-identification program can beconfigured to determine, before acquiring the image of the currentlypresented coating surface, the anticipated resolution of this imagegiven the current distance of the at least one camera from the presentedsurface area and/or given the current configuration of the at least onecamera. For example, the defects-identification program may becommunicatively coupled to the at least one camera and may request theanticipated resolution given the current distance and/or configurationdirectly from the at least one camera. According to other embodiments,the anticipated resolution is read from or computed based on aconfiguration file or other data source comprising the current settingsof the at least one camera.

According to other embodiments, the anticipated resolution is determinedbased on a reference object. A reference object of known size ispositioned on or next to the coating surface currently presented to theat least one camera. The at least one camera captures a preview-imagedepicting at least the reference object and determines the number ofpixels in the preview-image which depict the reference object. Thedetermined number of pixels in the preview-image which depict thereference object is the anticipated resolution of the at least onecamera in its present state. The anticipated resolution is compared withthe predefined resolution range to determine if the current, anticipatedresolution is within the predefined resolution range. For example, ifthe current distance between the at least one camera and the surface isdetermined to be below the predefined minimum resolution (and/or lieoutside of a predefined distance range) this can be used as anindication that the anticipated resolution will not enable thedefects-identification program to accurately identify the coatingdefects.

In case the anticipated resolution is identical to or above the minimumresolution, the user is enabled to capture the digital image via the atleast one camera manually or the at least one camera is caused tocapture the image automatically.

In case the anticipated resolution is below the predefined minimumresolution (and/or is outside of the predefined resolution range), thedistance between the at least one camera and the currently presentedcoating surface is modified such that the modified distance is withinthe predefined distance range, thereby ensuring that the anticipatedresolution of the camera at the modified position is identical to orabove the minimum resolution (and/or is within the predefined resolutionrange).

According to embodiments the image acquisition angle is adaptedanalogously. For example, a configuration file, a camera interfaceand/or or the computational identification of the shape of a referenceobject (which occurs only if viewed from a particular angle range)depicted in a preview image is used for determining the anticipatedimage acquisition angle given the current distance, position andorientation of the at least one camera.

The feedback signal may assist the user in positioning the cameracorrectly and/or to modify the setting of the image acquisition systemas to ensure appropriate image acquisition conditions. For example, thefeedback signal may show the user in which direction or at which anglehe has to move or rotate the camera in order to position it at asuitable position relative to the coating surface. In some examples, thefeedback is output via a GUI. In addition, or alternatively, thefeedback signal can comprise an acoustic alarm.

For example, a user or an application program or a configuration filemay select or determine the type of defects to be identified, e.g., viaa GUI or via a configuration file. The selection or determination of thedefect types to be examined and identified in the image may hencedetermine which image acquisition angles and/or distances are consideredallowable for identifying a particular type of defect. This may increaseaccuracy of the defect identification, because it was observed thatdifferent types of defects can be identified best at differentillumination and/or image acquisition conditions.

According to other embodiments, a corresponding feedback signal isgenerated and provided to an image acquisition unit of a facility forautomatically manufacturing and/or testing coating compositions. Thefeedback signal can comprise control commands causing the camera and/orthe light source comprised in the image acquisition unit toautomatically change their relative position to the coating surface, theillumination angle and/or the image acquisition angle such that theimage acquisition unit is enabled to acquire digital images of a coatingsurface currently comprised in the unit under conditions required by thedefect-identification program for proper image analysis.

For example, the feedback signal can be a signal for a human user forenabling the user to correct the position (distance and/or orientation)of the camera manually or semi-automatically. The feedback signal may inaddition or alternatively comprise a machine-command for enabling acontroller unit to correct the position of the camera automatically orsemi-automatically.

The feedback signal can be, for example, an overlay-image superposed ona preview image of the coating surface acquired by the at least onecamera, whereby the overlay-image comprises one or more GUI elementsindicating whether or not the at least one camera is in a positionsuitable for acquiring an image which can be successfully processed bythe defects-identification-program.

According to embodiments, the defects-identification program is selectedfrom a group comprising:

-   -   an application program, wherein the data processing system is a        stationary or portable data processing system, e.g. a        general-purpose data processing device, in particular a portable        telecommunication device, e.g. notebooks, smartphones, tablet        computers or other general-purpose data processing devices can        be used as portable data processing systems; the camera can be        an inbuilt, device-internal camera or an external camera, e.g. a        digital camera or camcorder; using general-purpose data        processing systems may have the advantage that no additional,        special-purpose hardware is required for enabling the staff of a        company to conduct an accurate and reproducible quality control        of coating compositions and/or coating surfaces manufactured by        the company;    -   an application program, the data processing system being a        portable or stationary device specially designed for quality        control of coating surfaces; for example, the specially designed        quality control device can comprise additional components such        as a camera and/or an illumination source whose position        relative to the coating surface can be controlled by the        defect-identification program; embodiments of the invention may        ensure that these devices are able to automatically identify and        characterize coating surface defects and the quality of the        examined coating surfaces in a reproducible and accurate manner;    -   an application program, the data processing system being a        high-throughput (HT) facility (also referred to as “high        throughput equipment—HTE) for the automated or semi-automated        manufacturing of coatings; in particular, the high throughput        facility can be a facility comprising an automated image        acquisition unit as described herein for embodiments of the        invention; using the defects identification program in the        context of an HT-facility may be particularly advantageous,        because HT-facilities are able to automatically manufacture and        test many different coating compositions and coating surfaces,        thereby creating large amounts of data which again can be used        for training a machine learning model to identify and/or predict        coating compositions having desired coating quality        characterizations; automatically generating and storing        qualitative and/or quantitative coating characterizations in the        context of an HT-facility may allow to consider coating defects        and coating quality characterizations in various big data        applications, in particular machine learning based predictions;        this was not possible based on manually created, subjective and        inconsistent quality labels;    -   a web application downloaded and/or instantiated permanently or        temporarily via a network; for example, a server can provide a        Java application implementing the defect-identification program        via the Internet; or    -   a program executed within a browser, e.g. a JavaScript program.    -   a server program instantiated on a server computer, the server        program being operatively coupled via a network connection to a        client program instantiated on a client data processing system        (e.g. a general purpose computer, a device specially designed        for quality control, or a smartphone); for example, the client        program can be a program configured to control one or more        cameras for acquiring one or more digital images of coating        surfaces; the acquired digital images are sent via the network        to the server program used for processing the images and output        the processing results including the characterization of the        coating surface via the network to the client program. The        client program can in addition be configured for displaying the        results provided by the server program.

A “portable device” can be, for example, a hand-held device such as asmartphone, but according to some examples, a portable device can alsobe a larger device up to a weight that can be carried by a human beingfor at least a few meters without the aid of technical means. Typically,such devices weigh less than 50 kg, in particular less than 40 kg.

Combinations of the above-mentioned embodiments are possible. Forexample, it is possible that the defect-identification program isimplemented as a client-server-system, wherein the client part isimplemented as the web application or the browser-executed program, isinstantiated on the data processing system and is responsible foracquiring a digital image in sufficient quality and in an appropriatecontext, and wherein the server part is instantiated on a remote servercomputer and is responsible for performing the image analysis foridentifying the coating defects and for providing the coating surfacecharacterization. According to another example, the client program andthe client data processing system can be part of a facility forpreparing and/or testing coating compositions, e.g. an HTE. The serverprogram, for example, may also be a (remote) part of the said HTE, maybe used as a server for multiple client programs belonging to multipledifferent HTEs and/or may not be part of an HTE. The server computersystem can be a monolithic computer system or a distributed computersystem, e.g. a cloud computer system.

According to embodiments, the defect-identification program comprises apredictive model having learned from training data in a training stepperformed by a machine learning program to recognize the predefinedpatterns. In particular, the machine learning program can be a neuralnetwork.

According to embodiments, the training data comprises multiple labeledtraining images of multiple coating surfaces. For example, the trainingimages can comprise digital images depicting coating surfaces obtainedby applying a coating composition on substrates of many different types(e.g. wood, plastic, metal, paper and the like), by applying a coatingcomposition according to many different coating application protocols(e.g. once or multiple times, at different temperatures, using e.g.spraying, spreading, painting or immersion technique, after havingperformed different types of pre-treatment protocols of the substrate,etc.). In addition, or alternatively, the training images can comprisedigital images depicting coating surfaces obtained by applying manydifferent coating compositions, whereby the different coatingcompositions have been obtained by combining different types and/oramounts of components and/or by combining the components according todifferent manufacturing process parameters (e.g. mixing duration, mixingtemperature, mixing speed, etc.) and/or different application processparameters. Hence, the training data may cover a huge, multidimensionaldata space covering a variety of different substrates, coatingcompositions, coating composition manufacturing parameters and coatingapplication protocol parameters. The method further comprises storingthe training data in a database. Initially, the labels of the trainingdata will be manually annotated. In later training steps, the trainingdata may be extended by additional digital images of coating surfaceswhich have been automatically labeled and which are preferably checkedor corrected by a human annotator. The labels preferably comprise apixel-based indication of the boundaries of coating defects, the type ofcoating defect, and one or more quantitative and/or qualitativecharacterizations of the coating surface which in some cases may beidentical to a comprise measures of the one or more defects depicted inthe respective training image.

According to embodiments, the machine learning program is a neuralnetwork or a set of neural networks comprising a region proposalnetwork. The region proposal network is configured to create anchors inan input image for making a proposal whether the anchor likely containsan object (one of the defect patterns). The “anchors” can be understoodas sub-regions of the input image having anchor sizes covering the sizesof the defect patterns to be detected.

For example, the average size of certain types of defects such asbubbles, foam-depressions and the like may already be known. The sizeranges of these defects can be used as the expected size range of thedefect patterns. The region proposal network is configured to create anduse for the detection of defects (e.g. “bubbles” and “craters”) anchorshaving sizes which cover the size ranges of the expected sizes of one ormore defect types to be identified. According to some embodiments,multiple different anchors sizes are defined by a user during training,e.g. anchors from a set of anchors having the sizes 8×8 pixels, 16×16pixels, 32×32 pixels, 64×64 pixels and 128×128 pixels can be used.

A “region proposal neural network” (RPN) as used herein is a neuralnetwork configured to operate on many different sub-regions of an inputimage referred to as “anchors” which are adapted to make “proposals”regarding the occurrence of an object in the sub-region.

According to preferred embodiments, the region proposal neural networkis a region proposal network comprised in the “Mask R-CNN” model asdescribed, for example, in “Mask R-CNN”, Kaiming He and Georgia Gkioxariand Piotr Dollár and Ross Girshick, 2017, eprint 1703.06870,arXiv:1703.06870. The “Mask R-CNN” is a flexible neural network-basedprogram configured for efficiently detecting objects in an image whilesimultaneously generating a high-quality segmentation mask for eachinstance. The Mask R-CNN program extends Faster R-CNN by adding a branchfor predicting an object mask in parallel with the existing branch forbounding box prediction. Mask R-CNN has four main outputs: For eachcandidate object (candidate defect type instance), a class label, ascore and a bounding-box is provided. In addition, an object mask can beprovided. Mask R-CNN adds only a small overhead to Faster R-CNN.Applicant has observed that the Mask R-CNN is particularly suited foridentifying coating surface defects of many different types as it iseasy to generalize to many different tasks in the same framework. MaskR-CNN can be downloaded via https://github.com/matterport/Mask_RCNN. Theuse of the Mask RCNN for instance segmentation of other object types isdescribed in the online article “Splash of Color: Instance Segmentationwith Mask R-CNN and TensorFlow” written by Waleed Abdulla Mar. 20, 2018and available online via the URLhttps://engineering.matterport.com/splash-of-color-instance-segmentation-with-mask-r-cnn-and-tensorflow-7c761e238b46.The Faster R-CNN approach is described by Shaoqing Ren and Kaiming Heand Ross Girshick and Jian Sun in “Faster R-CNN: Towards Real-TimeObject Detection with Region Proposal Networks”, 2015, eprint 1506.01497available via https://arxiv.org/abs/1506.01497.

According to one example, the neural network comprises a “backbone”network and an RPN. The “backbone” network can be a standardconvolutional neural network (typically, ResNet50 or ResNet101) that isapplied on the input image first and that serves as a feature extractor.The early layers of the “backbone network” detect low level features(edges and corners), and later layers successively detect higher levelimage features. The RPN is preferably trained to produce regionproposals based on feature vectors derived from the input image providedby the backbone neural network.

The operation of Mask R-CNN may roughly be summarized as follows:

-   -   the image is run through the “backbone network” to generate the        feature maps.    -   The RPN generates multiple Regions of Interest (RoIs) using a        lightweight binary classifier. It does this using anchors boxes        over the image. The “anchors” or “anchor boxes” can be        understood a set of bounding boxes covering a certain size range        to capture the scale and aspect ratio of specific defects types        to be detected. According to embodiments, the multiple different        anchors are processed in parallel on a CPU (central processing        unit) or GPU (graphical processing unit). This will greatly        increase processing speed for the detection of defect instances        of one or more different defect types. Anchor boxes also help to        detect multiple objects, objects of different scales, and        overlapping objects without the need to scan an image with a        sliding window that computes a separate prediction at every        potential position. This makes real time object detection        possible. For each of the anchors the region proposal network        predicts whether the anchor contains an object (foreground        class) or not (background class). For all foreground anchors in        addition a bounding box refinement is performed to better fit        the object (coating defect). These regions are also called        regions of interest (RoI).    -   For each RoI, a proposal is predicted. Each proposal is a        combination of a score of the RoI's probability of depicting a        particular type of object (e.g. 93% for a bubble defect) and        also the class/label of the object (e.g. “bubble defect”,        “cratering defect” or “background”).    -   The regions of interest are further refined by performing an        additional bounding box refinement step.

Additionally, segmentation masks are predicted for the positive anchors(regions of interest containing an object), being indicative of anidentified defect instance on the pixel level. For example, a Mask R-CNNhas been observed to be able to identify and characterize foam defectshighly accurately.

The above-mentioned steps describe how the already existing and/ortrained defect-identification program can be applied on new (test)images which do not comprise a label indicating the occurrence, type orextent of coating defects. In the following, embodiments of a method forgenerating and training the predictive model of the defectidentification program are described. The test phase and the trainingphase can be performed on the same or on different data processingsystems. For example, it is possible to train the model M1 on a firstdata processing system, integrate the trained model M1 in adefect-identification program which may comprise some additionalfunctions or modules, e.g. for interacting with a user and/or with afacility for manufacturing or testing coatings, and transfer thedefect-identification program to a second data processing system.

Training Phase of the Model (M1) of the Defect-Identification Program

According to embodiments, the method comprises performing the trainingstep on the training data, the training data comprising a set of labeleddigital training images of coating surfaces, the labels identifying thelocation/positions and/or type of defects in the training images. Thepredictive model is trained for recognizing the pattern by means of thelabeled training images using back propagation.

For example, the training images may respectively have assigned labelsindicating whether the image depicts one or more defect types which donot indicate the location and/or extent of the coating defects.Preferably, the training images comprise images of various coatingsurfaces being free of any defect, training images comprising defects ofa single defect type of varying extent, and/or training imagescomprising a mixture of defects of two or more different types.

The labels of the training data can comprise image class labels whichindicate the defect types(s) depicted in the image, if any, but are freeof positional information of the defects. In addition, or alternatively,the labels can comprise semantic defect type labels, objectidentification labels and/or defect type instance labels which provideinformation on the location of the defect types or defect type instanceson the pixel level or at the level of bounding boxes.

In addition, or alternatively, the training images according toembodiments of the invention have assigned a quantitative measure of oneor more defects depicted in the training images, e.g. the size and/orseverity of the defect or the number of bubbles, etc. These parametersand respectively assigned labels and quantitative measures are processedin the training step for enabling the predictive model to correlate theparameters with the defect patterns and with the measures of the coatingdefects of the coating surface.

According to embodiments, the defect identification program can comprisemultiple predictive models M1.1, M1.2, M1.3 respectively having learnedto correlate some defect-related image annotations with intensity orcolor patterns in a digital image. For example, the predictive modelM1.1 may be the region proposal network of the Mask-RCNN program havinglearned to correlate annotated defects and pixel patterns, and the modelM1.2 may be another neural network having learned to correlate defectmeasures and other parameters stored in association with the respectivedigital images with the annotated defects. The defect identificationprogram comprises the multiple predictive models M1.1, M1.2, M1.3 andintegrates and/or aggregates these models and their results forperforming the prediction.

Providing the defect-identification program in a machine learningprocess may have the advantage that the generated predictive model M1will have learned a plurality of highly complex interrelations betweenimage patterns and various defect types. Applicant has observed that itis very difficult and, in some cases, even impossible to integrate thesecomplex and largely hidden interrelations into a predictive model bymeans of one or more rules explicitly specified by a human.

According to embodiments, at least some of the training images have inaddition assigned parameters related to the coating compositions usedfor creating the coating surfaces depicted in the training images.

These additional, optional parameters assigned to the training imagesused for training the predictive model M1 (or one of multiple otherpredictive models M1.2 to be used by the defect identification program)can comprise one or more of:

-   -   an indication of one or more components of the coating        composition used for generating the coating surface depicting in        the training image;    -   an indication of an absolute or relative amount of one or more        of the components of the coating composition; for example, the        parameters may comprise a specification of the type and/or        amount of one or more defoamers comprised in the coating        composition used for generating the depicted surface; and/or    -   one or more manufacturing-process parameters, the        manufacturing-process parameters characterizing a process of        generating a coating composition, the process parameters for        example comprising mixing speed, the mixing temperature, and/or        the mixing duration of the coating composition; and/or    -   one or more application-process parameters, the        application-process parameters characterizing a process of        applying a coating composition on a substrate, the        application-process parameters in particular comprising the        amount of coating composition applied per area of the coating        surface, the type of substrate and/or or the type of application        equipment (e.g. machines or devices); the application process        parameters may also comprise parameters being indicative of a        method of pre-processing the substrate onto which the coating        composition is applied, e.g. drying, heating, cleansing or        otherwise preparing the substrate; and/or    -   system parameters of an imaging system used for acquiring the        training images, the system parameters being selected from a        group comprising type of light source(s) used for illuminating        the coating surface, brightness of the light source(s),        illumination angle, wavelength of the light source(s), type(s)        of one or more camera(s) used for acquiring the digital image of        the coating surface, image acquisition angle(s), position of the        one or more camera(s).

Training the predictive model to be integrated in thedefect-identification program on the above-mentioned context data may beadvantageous, because applicant has observed that the above-mentionedcontext parameters may all have an impact on the number and type ofdefects to be observed in the coating surface and hence have an impacton the coating surface quality. Annotating the training images with theabove-mentioned context data and/or the quantitative measure of thedefects ensure that the trained predictive model M1 is able to take intoaccount any factor that may have an impact on the type and extent ofcoating defects and on the quality characterizations of the coatingsurface.

Further Embodiments

In a further aspect, the invention relates to a computer-implementedmethod for providing a coating-composition-related prediction program,e.g. a composition-quality-prediction program and/or acoating-composition-specification-prediction program. The methodcomprises:

-   -   providing a database comprising associations of qualitative        and/or quantitative characterizations of coating surfaces in        association with one or more parameters selected from the group        comprising one or more of the components of the coating        composition used for producing the respective coating surface,        relative and/or absolute amounts of one or more of the said        components, manufacturing-process parameters of the coating        composition and/or application-process parameters used for        creating the coating surfaces;    -   training a machine learning model on the associations of the        coating surface characterizations with the one or more        parameters in the database for providing a predictive model        (e.g. predictive models referred herein as model “M2” or “M3”)        having learned to correlate qualitative and/or quantitative        characterizations of one or more coating surfaces with one or        more of the parameters stored in association with the respective        coating surface characterizations; and    -   providing a composition-quality-prediction program which        comprises the predictive model (M2), the        composition-quality-prediction program being configured for        using the predictive model (M2) for predicting the properties of        a coating surface to be produced from one or more input        parameters selected from the group comprising: one or more        components of a coating composition to be used for producing a        coating surface, relative and/or absolute amounts of one or more        of the said components, manufacturing-process parameters to be        used for preparing the coating composition and/or        application-process parameters to be used for creating the        coating surface; and/or    -   providing a composition-specification-prediction program which        comprises the predictive model (M3), the        composition-specification-prediction program being configured        for using the predictive model (M3) for predicting, based on an        input specifying at least a desired coating surface        characterization, one or more output parameters related to a        coating composition predicted to generate a coating surface        having the input surface characterizations, the one or more        output parameters being selected from the group comprising: one        or more components of the said coating composition, relative        and/or absolute amounts of one or more of the said components,        manufacturing-process parameters to be used for preparing the        coating composition and/or application-process parameters for        creating the coating surface; optionally, the        composition-specification-prediction program is configured for        receiving an incomplete coating composition specification and        for using the specification for limiting the solution space of        the predicted output parameters.

Embodiments of the invention may have the advantage that acomposition-quality prediction program is provided which is able toautomatically predict quality characterizations of a particular coatingcomposition as a function of the type and/or amount of the components ofthe composition and optionally as a function of further parameters. Thismay tremendously accelerate the process of testing and identifying acoating composition which is able to provide a coating surface havingthe desired quality characterizations. Contrary to prior art approacheswhich are based on human experience and a typical large number ofcoating compositions manufactured and tested at the workbench,embodiments of the invention may allow accurately predicting, whether ornot a particular coating composition will have the desired coatingsurface characterizations or not. This may tremendously accelerate theprocess of identifying a suitable coating composition and reduce thecosts associated with reagents, machines and consumables required toperform this identification process.

While the predictive model M1 of the defect-identification program ispreferably obtained by performing a machine learning step on manuallyannotated training images and has learned to correlate pixel patternswith defect-type characterization measures and coating surfacecharacterizations, the predictive model M2 of the composition-qualityprediction program can be trained on training data which may comprisebut preferably do not comprise digital images. The purpose of thepredictive model M2 is to predict coating surface characterizations, inparticular quality-related characterizations, as a function of the oneor more components and optionally also context parameters of a coatingcomposition.

According to embodiments, the method comprises providing a plurality ofimages respectively depicting a coating surface. The depicted coatingsurfaces are made by applying multiple different coating compositions onrespective surface samples. At least some of the depicted coatingsurfaces respectively have one or more coating defects of one or moredifferent defect types. The method comprises applying adefect-identification program on the images for recognizing patterns inthe images, for obtaining the measures of the coating defectsrepresented by the identified patterns and for computing a qualitativeand/or quantitative characterization of the coating surfaces depicted inthe images. The method further comprises storing the qualitative and/orquantitative characterizations of the coating surfaces in associationwith one or more parameters related to the coating composition used forcreating the coating surface comprising these defects in the databasefor providing the training data for the predictive model (M2, M3). Forexample, these parameters can be selected from the group comprising oneor more components of the said coating composition, relative and/orabsolute amounts of one or more of the said components,manufacturing-process parameters to be used for preparing the coatingcomposition and/or application-process parameters for creating thecoating surface.

This may be advantageous as the automated defect identification andcoating surface characterization may allow to annotate a huge number ofimages automatically in a reproducible and comparable manner. Providinga large, unbiassed training data set may ensure that the model M2 havingbeen trained on this training data set is able to predict the coatingcomposition properties, in particular the quality of a coating surfaceobtained by applying the composition on a substrate, accurately.

According to embodiments, the defect-identification program is thedefect-identification program specified in any one of the embodiments orexamples described herein.

According to embodiments, the qualitative and/or quantitative measureshave assigned parameters considered in the machine learning step forgenerating the predictive model (M2, M3) to be used by thecomposition-quality-prediction program or by thecomposition-specification-prediction program. The parameters comprise:

-   -   an indication of one or more of the components of the coating        composition; for example, an indication of a component can        specify a particular chemical compound, e.g. a particular        pigment or a particular defoamer, or a substance class, e.g.        “solvent-based coating medium”;    -   an indication of the absolute or relative amount of one or more        of the components of the coating composition; for example, the        indication may specify the relative amount of two defoamers;    -   one or more manufacturing-process parameters, the        manufacturing-process parameters characterizing a process of        generating a coating composition, the process parameters for        example comprising mixing speed and/or mixing duration of the        coating composition; and/or    -   one or more application-process parameters, the        application-process parameters characterizing a process of        applying a coating composition on a substrate, the        application-process parameters in particular comprising the        amount of coating composition applied per area of the substrate,        the type of substrate and/or or the type of application        equipment.

A data file or data record comprising the above-mentioned parameters isalso referred to as “specification” of a coating composition. Aspecification is complete if it specifies the nature and amount of eachcomponent and specifies all further information needed in the respectiveapplication context in order to prepare a coating composition and acorresponding coating surface. Typically, the specification isincomplete: for example, the nature and/or amount of at least some ofthe components may not be specified or somecoating-composition-manufacturing process parameters and/orcoating-composition-application-parameters may not be specified.

According to embodiments, the method comprises using thecomposition-quality-prediction program for predicting the properties ofa coating composition. The coating composition can be a paint, avarnish, a printing ink, a grinding resin, a pigment concentrate orother substance mixture for coating a substrate.

According to embodiments, the method comprises using thecomposition-specification-prediction program for predicting one or moreof the above-mentioned parameters (nature and/or amount of coatingcompositions, manufacturing-process parameters, application-processparameters) based on a characterization of a desired coating surfaceprovided as input. These parameters can be used for completing and/oroptimizing a specification of a coating composition.

According to preferred embodiments, a combination of both models M2, M3is used for supplementing and/or optimizing a coating compositionspecification. For example, the composition-quality-prediction programusing predictive model M2 may receive as input a specification of acoating composition which in addition comprises some manufacturingand/or application process parameters. Based on the receivedspecification, the composition-quality-prediction program may predictthat the coating surface may comprise many bubble defects. Then, theuser may input into the composition-specification-prediction programusing the model M3 that a coating composition specification shall bepredicted which minimizes bubble defects, whereby the coatingcomposition specification can be provided to thecomposition-specification-prediction program as additional input andconstraint for the prediction. The model M3 will determine that amodification of the ratio of the two defoamers will result in areduction of bubble defects and may output a modified version of theinput specification, the modified specification comprising a modifiedratio of the two defoamers.

Coating compositions are complex mixtures of raw materials. Commoncompositions or recipes or formulations for coating compositions containabout 20 raw materials, hereinafter also referred to as “components”.These compositions consist, for example, of raw materials selected fromamong solids, such as pigments and/or fillers, further binders,solvents, resins, hardeners and various additives, such as thickeners,dispersing agents, wetting agents, adhesion promoters, defoamers,surface modifiers, levelling agents, catalytically active additives suchas dryers and catalysts and specially effective additives such asbiocides, photoinitiators and corrosion inhibitors.

Up to now, new compositions, formulations and reformulations withcertain desired properties have been specified on the basis of empiricalvalues and then prepared and tested. The composition of a newcomposition that fulfils certain expectations in terms of its chemical,physical, optical, haptic and other measurable properties and inparticular in terms of the quality/defectiveness of a coating surfaceobtained by coating a substrate with the composition is hardlypredictable even for an expert due to the complexity of the interactionsand due to a lack of high quality data suitable for use in amachine-learning training step. Due to the diversity of the interactionsof the raw materials and process parameters among each other and theassociated large number of failed experiments, this procedure is bothtime-consuming and costly.

Thus, there are currently very narrow limits to both human-made andcomputer-aided assessment and prognosis regarding the components of acomposition with desired properties. This is particularly true forcomplex compositions with many relevant properties and many components,as is the case with coating composition such as paints, varnishes,printing inks, grinding resins, pigment concentrates or other coatingmaterials, since the components interact with each other in a complexway and determine the properties of the corresponding coatingcomposition. The identification of one or more defoamers being alone orin combination able to minimize the bubble defect generation has beenobserved to be particularly challenging.

At present, new coating surfaces must first be produced in a reallaboratory environment and their properties must then be measured inorder to assess whether a new coating composition has certain requiredproperties. Although there are already approaches for the automaticprediction of properties of chemical substances, the creation of atraining data set of sufficient size and quality is often even morecomplex than directly producing and testing the composition in question.The development of new compositions in the field of paints, varnishes,printing inks, grinding resins, pigment concentrates or other coatingmaterials is particularly complex and requires a lot of time.

According to embodiments, the using of thecomposition-quality-prediction program comprises:

-   -   providing each of the specifications of a plurality of different        candidate coating compositions as input to the        composition-quality-prediction program;    -   predicting, by the composition-quality-prediction program, for        each of the candidate coating compositions the quality of a        coating surface generated by applying the candidate coating        composition on a substrate, the prediction being performed as a        function of the specification of the candidate coating        composition;    -   selecting a candidate coating composition specification in        dependence on the respectively predicted measures; for example,        the selection can be performed manually such that the one of the        candidate coating composition specification is selected whose        predicted coating surface characteristics best meets a quality        requirement; according to other embodiments, one or more        parameters being indicative of one or more desired surface        characterizations used as quality criteria are provided as input        to the composition-quality-prediction program, thereby enabling        the composition-quality-prediction program to perform this        selection step automatically in dependence on the input quality        criteria, and    -   outputting the selected specification as a recommended coating        composition predicted to have the highest quality; and/or    -   inputting the selected candidate coating composition        specification to a processor which controls a facility for        producing and/or testing compositions for coating compositions,        wherein the processor drives the facility to produce the input        coating composition.

For example, the prediction is generated by a computer system connectedto the facility for producing and/or testing compositions for coatingcompositions. The facility can be a HT-facility.

According to embodiments, the method further comprises:

-   -   receiving an incomplete specification of a coating composition;    -   manually or automatically generating a set of specifications of        candidate coating compositions; the set of candidate        compositions are different versions of the received incomplete        coating composition; the creation of the specifications of the        candidate coating compositions comprises:        -   a) supplementing the specification of the incomplete coating            composition with one or more further components; and/or        -   b) supplementing the specification of the incomplete coating            composition with different absolute or relative amounts of            the one or more further components and/or with different            absolute or relative amounts of the already specified            components; and/or        -   c) supplementing the specification of the incomplete coating            composition with one or more            manufacturing-process-parameters characterizing the process            of manufacturing the candidate coating composition; and/or        -   d) supplementing the specification of the incomplete coating            composition with one or more application-process-parameters            characterizing the process of applying the candidate coating            composition on a substrate.

For example, the received incomplete specification may specify that the(desired) coating composition to be produced should comprise/be based onan organic solvent but the incomplete specification does not specifywhich one of a plurality of available solvents should be used. Thecandidate compositions could be created such that different organicsolvents (but not a water-based solvent) are used as the“solvent”-component.

According to another example, the received incomplete specification mayspecify that the (desired) coating composition should comprise twoparticular defoamers (e.g. TEGO Foamex 810 and TEGO Wet 285, Co. Evonik)but does not specify their absolute or relative amounts. The candidatecompositions could be created such that different amount ratios of thetwo defoamers are used.

According to another example, the incomplete specification may specifythat a particular pigment or pigment combination should be used but issilent about the amount of the pigment or only provides an amount rangeof the pigment(s). The candidate compositions could be coatingcompositions differing from each other in respect to the amount of thepigment(s).

According to another example, the incomplete specification may specifyall or at least most of the components of the coating composition andtheir respective amounts but is silent about context parameters, inparticular manufacturing process parameters such as mixing time, mixingduration, mixing temperature etc. and/or coating application processparameters (spreading, spraying, painting, immersing, pre-treating thesubstrate, temperature, ventilation, etc.). The candidate compositionscould be coating compositions differing from each other in respect tothe parameter values of one or more of the above-mentioned contextparameters.

According to embodiments, the further data used for supplementing thecandidate coating composition specifications in the respective cases a),b), c) and/or d) is used as input by the coatingcomposition-quality-prediction program for performing the prediction.

Embodiments of the invention may be beneficial for manufacturers ofcoating additives as well as manufacturers of coating compositions formultiple reasons: manufacturers of coating compositions may predictproperties of a plurality of various candidate compositions incombination with various context parameter values without actuallymanufacturing and testing these coating compositions.

However, embodiments of the invention may also be useful for themanufacturers of individual coating composition components such assolvents, binders, pigments, foaming agents, defoamers, rheologicaladditives, flame retardants, dispersing agents, and the like. Forexample, the manufacturer of a defoamer may receive an order for adefoamer to be used in a coating composition for wooden materials from amanufacturer of a coating composition. The received specification mayindicate that the coating is to be applied on wooden materials and isbased on water as a solvent, but detailed specification of the coatingcomposition is not disclosed for confidentiality reasons. Nevertheless,the manufacturer of the defoamer would like to identify the type andamount of one or more defoamers, ideally a defoamer mixture, and theamount of the solvent which in combination are best suited to provide acoating for wooden materials with a high coating surface quality. Inorder to achieve this, the employees of the additive manufacturer or asoftware program generates a plurality of candidate coating compositionspecifications which differ from each other in respect to the type,mixture thereof and/or amount of the defoamer and/or in respect to theratios of the defoamers used. Each candidate coating compositionspecification comprises an indication that the coating is to be used tocoat wooden materials. Then, the composition-quality-prediction programuses each of the candidate coating composition specifications as inputfor predicting a characterization of the respective candidate coatingcomposition, in particular a characterization being indicative of thequality of a coating surface generated by applying the candidate coatingcomposition on a wooden substrate. As the composition-quality-predictionprogram has learned to correlate a plurality of types and/or amounts ofcomponents and optionally also context parameters with coating surfacecharacterizations, the composition-quality-prediction program is able toidentify a suitable combination of defoamers. The manufacturer of thedefoamer can provide the defoamer mixture to the manufacturer of thecoating composition together with a recommendation regarding therelative or absolute amounts of the defoamer and the solvent and/ortogether with a recommendation for process parameters relating to thecreation, storing and/or application of the coating composition.

Embodiments wherein the composition-quality-prediction programs isconfigured to generate a specification of a recommended coatingcomposition which in addition specifies one or more of theabove-mentioned parameter types a)-d) may be advantageous as applicanthas observed that not only the composition but also process parametershave a tremendous impact on the properties and quality of a coatingsurface. However, due to the huge number of combinatorically possiblecombinations of components and process parameters, it was hitherto notpossible to evaluate all combinations which might correspond to ahigh-quality coating surface.

The above described embodiments are based on providing a plurality of(hypothetical) candidate coating composition specifications, predictingthe respective coating surface quality and selecting the one of thecandidate coating compositions whose predicted surface properties appearto be most desirable. However, alternative approaches may use thelearned correlations between the coating surface characteristics and thecoating composition related parameters which have been incorporated inthe predictive model M3 more directly for identifying promising coatingcompositions:

According to embodiments, the using of the coatingcomposition-specification-prediction program comprises:

-   -   providing at least a specification of a desired coating surface        characterization as input to the        composition-specification-prediction program; and    -   predicting, by the composition-specification-prediction program,        a specification of a coating composition adapted to provide a        coating surface having the desired surface characteristics,        wherein the specification comprises parameters selected from a        group comprising: one or more coating composition components,        absolute or relative amounts of one or more of the coating        composition components, manufacturing process parameters and/or        application process parameters.

According to preferred embodiments, the method further comprisesoutputting the predicted specification of the coating composition to ahuman and/or inputting the specification of the selected candidatecoating composition to a processor which controls a facility forproducing and/or testing compositions for coating compositions, whereinthe processor drives the facility to produce the input coatingcomposition.

In addition to the desired surface characterizations, additionalconstraints may be provided as input. The constraints may consist of anincomplete, rough coating composition specification being indicative ofsome components or component substance classes and some absolute orrelative amounts. The constraint may be that any alternative componentsuggested must belong to the same substance class or any alternativeamount must not deviate from the amount provided in the constraint bymore than a maximum threshold value. Manufacturing process parametersand/or application process parameters may also be provided asconstraints.

According to embodiments, the method is performed on a computer systemoperatively coupled to an automated surface coating and imageacquisition unit (ACAIA unit). The ACAIA unit comprises one or morecameras and optionally one or more light sources. The ACAIA unit is acomponent of a facility for producing and/or testing coatingcompositions or is operatively coupled to the facility by automatedtransport means, e.g. for automatically transporting a coated substratesample to and from the ACAIA unit. The method further comprises sendingone or more control commands to the facility. The control commands causethe facility to:

-   -   transport the produced coating composition to the ACAIA unit and        apply the transported coating composition on a substrate;        alternatively, the coating process is performed in a separate        coating unit of the facility and the facility is configured to        transport the already coated substrate sample to the ACAIA unit;    -   positioning the coated surface and the camera relative to each        other within a predefined distance and/or predefined angle        range; for example, the predefined distance and/or angle range        can be distances and/or angles specific for one or more types of        surface defect types to be detected in the digital images to be        acquired; the indication of the defect types of interest may be        provided with the control commands or may be specified in a        configuration of the ACAIA unit; and    -   after the positioning, causing the camera of the ACAIA unit for        acquiring a digital image of the coating surface; and    -   returning the acquired image to the computer system.

This may be advantageous as a fully automated system for predicting,manufacturing, and testing a coating composition in the context of anautomated manufacturing facility, in particular a HT-facility, isprovided. The data obtained in the image-analysis based testing step maybe used for extending the training data of thecomposition-quality-prediction-program and/or of thecomposition-specification-prediction program, and for re-training themodel (M2, M3) of the composition-quality-prediction-program or of thecomposition-specification-prediction program on the extended trainingdata for obtaining an improved version of the model M2+, M3+.

According to embodiments, the method comprises:

-   -   applying a defect-identification program on the returned image        for identifying measures of one or more defects in the coating        surface depicted in the returned image and for computing a        qualitative and/or quantitative characterization of the said        coating surface; for example, the defect-identification program        may initially be obtained by training a machine-learning model        on manually annotated images of coating surfaces;    -   storing the qualitative and/or quantitative characterizations in        the database for providing an extended database; and    -   re-training the predictive model (M2, M3) on the associations of        the characterizations and the coating composition associated        parameters (components, absolute and/or relative component        amounts, manufacturing process parameters and/or application        process parameters) in the extended database for providing an        improved version of the predictive model; and    -   replacing the predictive model (M2) of the        composition-quality-prediction program and/or the predictive        model (M3) of the composition-specification-prediction program        with the improved version of the predictive model.

Hence, in the context of an automated facility for manufacturing andtesting a coating composition, thecomposition-quality-prediction-program may be used for predicting manypromising candidate compositions, the facility may be used formanufacturing and testing the real properties of these candidatecompositions, including the coating surface quality, and the obtainedtest data may be used for improving the predictive model M2 of thecomposition-quality-prediction program and/or the predictive model M3 ofthe composition-specification-prediction program. Hence, in the courseof time, the integrated system comprising the facility and thecomposition-quality-prediction-program (and/orcomposition-specification-prediction) have the potential of mutualenhancement and improvement.

According to embodiments, the training process involves an activelearning module. The method further comprises:

-   -   a. performing the training of the predictive model on the        associations of the measures and the coating components in the        database for providing the trained predictive model (M2, M3),        where a loss function is minimized for the training,    -   b. testing to determine whether the value of the loss function        obtained for the predictive model meets a specified criterion,        -   whereby selectively, in the event that the criterion is not            met, the following steps are carried out:            -   i. selection of a candidate coating composition                specification from a plurality of candidate coating                composition specifications by the active learning                module, the selected candidate composition specification                specifying the one of the candidate compositions                determined to provide the highest learning effect of the                predictive model (M2, M3) regarding the correlation of                qualitative and/or quantitative coating surface                characterizations and one or more parameters selected                from a group comprising coating components, component                amounts, manufacturing process parameters and/or                application process parameters of the coating                composition;            -   ii. driving the facility by the computer system for                automatically producing the candidate coating                composition in accordance with the selected                specification, for automatically applying the produced                candidate composition on a substrate and for                automatically acquiring an image of the surface with the                applied coating composition;            -   iii. applying the defects-identification program on the                image acquired in step ii for computing and storing a                qualitative and/or quantitative characterization of the                coating surface depicted in said image in the database,                thereby extending the database;            -   iv. re-training of the predictive model (M2, M3) on the                extended database for providing an improved version of                the predictive model,            -   v. repeated execution of step b using the improved                version of the predictive model,    -   c. replacing the predictive model (M2) of the        composition-quality-prediction program with the improved version        of the predictive model and/or replacing the predictive model        (M3) of the composition-specification-prediction program with        the improved version of the predictive model.

The use of an active learning module may have the advantage that thelearning effect during the training or re-training of the model M2 isimproved and accelerated: the active learning module is adapted toidentify the ones of the candidate coating composition the testing andstoring of whose properties would allow improving the quality of thepredictive model M2 the most.

Using of the adaptive learning module may be beneficial in the contextof a situation in which obtaining manufacturing and empirically testinga coating composition is expensive. In such a scenario, the activelearning module can actively identify the most promising candidatecompositions and can actively query for manufacturing and testing theseidentified candidate compositions. Since the active learner chooses themost promising candidate composition, the number of compositions to bemanufactured and tested to learn a concept can hence be much lower thanthe number required in normal supervised learning.

According to embodiments, active learning is used during the training ofthe predictive model M1 to be used for identifying coating defects indigital images of coating surfaces. In this case, the active learneridentifies a subset of one or more of a plurality of unlabeled testimages of coating surfaces whose manual annotation would provide thehighest learning effect. The active learning module prompts a user tomanually annotate (assign labels) to each image of the subset, wherebythe label indicates type and location of the coating defect depictedtherein. These one or more additionally labeled images are added to thetraining images, thereby extending the training data. The predictivemodel M1 is re-trained on the extended training data, thereby providingan improved, more accurate version of the predictive model Ml. Then, theoutdated predictive model in the defect-identification program isreplaced by the improved version of the model.

For example the active learning module can be part of the machinelearning program used for training or retraining the predictive model(M2, M3). The active learning module can likewise be a separate softwarecomponent operatively coupled to the predictive model and amachine-learning environment. The active learning module is operable toaccess the predictive model, evaluate the level of insecurity of aprediction generated by this predictive model, and initiate are-training of this predictive model based on a training data set.

According to embodiments, the active learning module is operativelycoupled to the facility for preparing and testing coating compositionsand is configured to drive this facility automatically. For example, theautomated driving of the facility may comprise identifying, by theactive learning module, the one of a plurality of candidate coatingcomposition specifications for which the predictive model generatespredictions with the highest uncertainty score. These are the coatingcomposition which provide the strongest learning effect. The facility iscontrolled to prepare and test the coating composition selected by theactive learning module. The active learning module receives theempirical properties of the newly prepared coating composition(including e.g. surface defects) and uses the data for retraining andimproving the predictive model.

According to some embodiments, the active learning module is configuredfor generating a GUI which displays to the user a plurality ofselectable GUI elements which represent different candidate coatingcompositions which have been selected by the active learning module.

The active learning module (or any software program comprising theactive learning module, e.g. the machine-learning framework) can be partof a control software configured to monitor and/or control the operationof the framework.

For example, the active learning module can be run on a data processingsystem 120.1, 120.2 comprising the machine-learning software (alsoreferred to as machine learning framework) as depicted, for example, inFIG. 7. The active learning module can also be run on a computer system924 operatively coupled to the facility for preparing and testingcompositions as depicted, for example, in FIG. 9. Likewise, the activelearning module can be an integral part of the facility and can be run,for example, on an integral control computer 946 of the facility. Thedata processing systems on which the active learning module isinstantiated can be, for example, a standard computer system, a server,or a portable computer system or a device-internal control computersystem. The said data processing system can also comprise software forautomatically generating candidate coating composition specificationsand/or may comprise an interface for receiving the specification from aremote source.

Using an active learning module in the context of the preparation andtesting of coating compositions may be particularly advantageous as itcan strongly reduce the numbers of candidate coating compositions and/orthe number of application methods which need to be empirically preparedand tested in order to obtain a prediction model with sufficientaccuracy. The number of components of a coating composition typicallystrongly exceeds the number of components of other substance mixturessuch as e.g. alloys, and the number of different application scenarios(temperature, moisture, etc.) is also huge. Using an active learningmodule for only selecting the ones of a plurality of candidate coatingcompositions which will provide the strongest learning effect, anddriving a facility to prepare and test these compositions for providingadditional training data may be particularly advantageous in the contextof the preparation of coating compositions.

The active learning module can be implemented, for example, based onexisting active learning libraries such as the phyton package libact ormodal, an active learning framework for Python3.

In a further aspect, the invention relates to a system comprising:

-   -   a facility for producing and testing compositions for paints,        varnishes, printing inks, grinding resins, pigment concentrates        or other coating materials, where the facility comprises at        least two workstations, where the at least two workstations are        connected to one another via a transport system on which        self-propelled transport vehicles are able to run for        transporting the components of the composition and/or of the        composition produced between the workstations, and    -   a computer system configured to perform the method of any one of        the embodiments described herein.

In a further aspect, the invention relates to a computer programconfigured to perform the method of any one of the embodiments describedherein.

In a further aspect, the invention relates to a defect-identificationprogram provided by performing a method of providing and using thedefect-identification program as described herein for embodiments of theinvention.

In a further aspect, the invention relates to a distributed dataprocessing system comprising a monolithic or distributed data processingsystem and a camera. The distributed or monolithic data processingsystem comprises a defect-identification program as described herein forembodiments of the invention adapted to identify coating defects indigital images of coating surfaces acquired by the camera. For example,the data processing system can be a monolithic data processing system,e.g. a general-purpose data processing system such as a computer or asmartphone or can be a surface-quality-control-related-special-purposedevice. Alternatively, the data processing system can be a distributeddata processing system, e.g. a cloud system or a client-server systemcomprising a server computer and one or more client data processingsystems.

In a further aspect, the invention relates to a composition-qualityprediction program provided by performing a method of providing andusing the composition-quality prediction program as described herein forembodiments of the invention.

In a further aspect, the invention relates to acomposition-specification-prediction program provided by performing amethod of providing and using the composition-specification-predictionprogram as described herein for embodiments of the invention.

In a further aspect, the invention relates to a coating-compositionmanufactured in accordance with a specification of a compositionprovided by performing a method of any one of the embodiments forproviding a specification of a coating composition described herein. Inparticular, the specification can be computed according to embodimentsof the invention by the coating-composition-specification-predictionprogram based on a desired coating surface characterization provided asinput using a trained model (M3).

In a further aspect, the invention relates to a specification of acomposition provided by performing a method of any one of theembodiments for providing a specification of a coating compositiondescribed herein.

In a further aspect, the invention relates to a volatile or non-volatiledata storage medium comprising computer-interpretable instructionsimplementing the defects-identification program, thecomposition-quality-prediction program and/or thecomposition-specification-prediction program.

In a further aspect, the invention relates to a volatile or non-volatiledata storage medium comprising the above-mentioned specification, thedata storage medium being operatively coupled via an interface to afacility, the facility being configured for producing coatingcompositions in accordance with one or more specifications stored in thedata storage medium.

A “coating surface” as used herein is a surface of a substrate havingbeen coated one or multiple times with a coating composition. Forexample, the coating composition can be applied by spreading or sprayingor painting the coating composition onto the substrate, by immersing atleast one surface of the substrate in the coating composition, or byother coating approaches.

A “coating defect” or “coating surface defect” as used herein is anyoptically detectable deviation of a coating surface from a typical ordesired appearance of the coating surface. An image defect resultinge.g. from dust on an optic lens of the camera is not a coating defect.

A “program” as used herein is a piece of software, e.g. an applicationprogram or a module or function of an application program, or a script,or any other kind of software code that is executable by one or moreprocessors, e.g. CPUs or GPUs. A program can be, for example, anapplication program. Application programs, in particular programsdesigned for execution on portable devices, can be implemented in theform of an “app”.

A “defect-identification program” as used herein is a software programor software module configured to analyze digital images forautomatically identifying one or more coating defects and for computinga coating surface characterization as a function of the identifiedcoating defects in the surface depicted in the image.

A “composition-quality prediction program” as used herein is a softwareprogram or software module configured to receive a (complete orincomplete) specification of a coating composition and is configured topredict one or more properties of the coating composition as a functionof the data specified in the specification. The properties of thecoating composition can comprise an indication of the quality of thecoating composition, e.g. an indication of the quality of a coatingsurface obtained by applying the coating composition on a substrate. Thequality indication may be, for example, the likelihood of generatingcertain types of coating defects.

A “composition-specification prediction program” as used herein is asoftware program or software module configured to receive a desiredcoating surface characterization as input parameter(s). Optionally, thecomposition-specification prediction program can be configured toreceive one or more further input parameters such as an incompletespecification of a coating composition for limiting the solution spaceof the prediction. The composition-specification prediction program isconfigured to predict one or more of the following output parameters asa function of the received input data: one or more components, one ormore absolute or relative component amounts, one or more manufacturingprocess parameters and/or one or more application process parameters.

A “measure” of a defect as used herein is any qualitative orquantitative parameter value or set of parameter values beingdescriptive of a property of a coating defect. Examples for aqualitative defect measure is a defect type label such as “bubbledefect”, “delamination defect”, “foam defect” and the like. Examples fora quantitative measure of a coating defect are “% of coating surfacecovered by the defect”, “number of bubbles in an image”, “numbers ofbubbles per area for the defect”, “average size of bubbles”, and thelike. A measure of a defect can be a single parameter value or a set ofparameter values, e.g. property of an individual defect instance or aproperty derived from multiple defect type instances, e.g. an averagebubble defect instance diameter or a histogram being indicative of thesize distribution of multiple defect instances, e.g. bubbles. A measureof a defect can be obtained e.g. by analyzing the dimensions or otherproperties of the identified defects (e.g. based on surrounding boundingboxes and/or based on individual pixels). A measure of a defect can be aparameter value, or a set of parameter values obtained by applyingarithmetic operations, e.g. for obtaining an average diameter, or bycomplex statistical calculations, e.g. a mathematical formulacharacterizing a distribution of the size and positional distance ofbubbles in a coating surface, a histogram, or the like.

A “characterization of a coating surface” as used herein is anyqualitative or quantitative parameter value or set of parameter valuesbeing descriptive of the quality of a coating of a surface. The coatingsurface quality depends on the measures of one or more defects comprisedin the coating surface. For example, a characterization of a coatingsurface can be identical to qualitative and/or quantitative measures ofone or more defects comprised in the coating surface. According to otherexamples, the characterization of the coating surface is obtained byaggregating the measures of multiple defects in the coating surface. Forexample, in case the coating surface comprises three bubble defects BD1,BD2, BD3, the aggregated coating surface characterization can be thefraction of the depicted surface covered by the totality of theidentified bubble defects, e.g. “5% bubble defect area”. The aggregationfunction can be, for example, an arithmetic or geometric mean (e.g. forcomputing average bubble diameters), a sum, a product, or more complexarithmetic and/or statistic aggregation functions. A characterization ofa coating surface can comprise a combination of two or more qualitativeand/or quantitative characterizations. For example, the characterizationcan be a combination of an indication of the defect type with an extentof the defect, e.g. “coating surface with a severe bubble defects”, or“coating surface with medium-grade bubble defects” or “coating surfacewith minor bubble defects”. The extent may be provided in the form of anumerical value, e.g. % of the area covered by the defects, or anindication of a value range, a histogram, or the like. Acharacterization of a coating surface can comprise a combination ofqualitative and/or quantitative characterizations obtained for each oftwo or more different types of defects respectively.

The term “composition” or “coating composition” as used herein is acomposition which comprises two or more raw materials ('components')from which the composition is formed and which is to be applied on asubstrate to provide a coating surface. When in the context of thisapplication reference is made to the production or testing of acomposition by an automated facility, this is to be understood that thecoating composition is produced according to information on the natureand/or amount of the components.

The term “candidate composition” or “candidate coating composition” asused herein is a coating composition which has not yet been preparedand/or tested and whose properties are therefore at least partiallyunknown at the moment when the candidate coating composition isspecified. For example, a candidate composition may be a compositionthat has been manually or automatically specified but which has not yetbeen prepared and tested. Accordingly, the properties of thiscomposition are not known.

A “specification of a composition” as used herein is a data setcomprising parameters related to a coating composition. For example, theparameters can indicate the identity and/or substance class of some orall components to be combined for manufacturing a coating composition.Optionally, the specification may comprise additional parameters, e.g.the relative or absolute amounts or amount ranges of the respectivecomponents, coating composition manufacturing process parameters,coating application process parameters, and the like. These parametersmay specify how the components have to be processed and/or mixed forobtaining the composition and/or how the composition is to be applied ona substrate to obtain a particular coating surface. The specification ofa composition may be complete or incomplete. For example, somespecifications may only indicate the type of a component (e.g.organic-solvent based solvating agent) but not the exact identity and/oramount of the component. The “specification” of a composition may beprovided in various forms, e.g. as printout, as file, e.g. an XML file,an object of an object-oriented programming language, as JSON file orthe like.

A “coating composition manufacturing process parameter” or“manufacturing process parameter” as used herein is a parameter beingindicative of properties of a process of processing and/or combining thecomponents to form a coating composition. Examples are mixing duration,mixing speed, mixing temperature, sequence of mixing components,equipment used for mixing or otherwise preparing the coating compositionor the like.

A “coating composition application process parameter” or “applicationprocess parameter” as used herein is a parameter being indicative ofproperties of a process of applying a coating composition on asubstrate. Examples are indications of the application technique(spraying, spreading, painting, immersing, etc.), the duration of theapplication (e.g. immersion time), number of repeated applications,pre-treatment steps of the substrate (drying, priming, cleaning,heating, etc.), the equipment used for applying the coating compositionand/or the nature of the substrate (wood, plastic, metal, cardboard,etc.).

A “known composition” as used herein is a composition whose properties(e.g. coating surface characterizations, rheological properties,elasticity, shelf-life etc.) are known at the time of training of aneural network to the person conducting the training. For example, theknown composition may have been produced for a customer several monthsor years ago and the properties of that product have been measuredempirically. The measurement does not necessarily have to have beencarried out by the operator of the laboratory which now determines thepredictive composition, but may also have been carried out and publishedby other laboratories, so that in this case the properties are takenfrom the specialist literature. Since a composition according to theabove definition also contains formulations as a subset, the “knowncompositions” according to the embodiments of the invention may alsocontain “known formulations” or be “known formulations”.

The term “defoaming” is frequently used to describe the removal of gasbubbles from the composition or the coating. However, in certain casesthe terms “defoaming” and “air release” should be differentiated. First,the gas bubbles need to reach the surface. Removing the foam bubblesthat are at the surface is called defoaming (in the strict sense).Defoamers (in the strict sense) are only effective at the surface wherethey eliminate the air bubbles located there. In comparison, air releaseagents take effect throughout the coating film. In the present case, theterm “defoamers” is used broadly and covers both the defoamers in thestrict sense and the air release agents.

A “database” as used herein is any volatile or non-volatile data storagemedium in which data, in particular structured data, is stored. Thedatabase can be one or more text files, spreadsheet files, a directoryin a directory tree, or a database of a relational database managementsystem (DBMS) such as MySQL or PostgreSQL.

A “loss function” of a prediction problem as used herein is a functionthat is used in the training of a predictive model (e.g. a model of aneural network) by a machine learning program for training and improvingthe model. The loss function outputs a value whose magnitude gives anindication of the quality of the predictive model, whereby the lossfunction is minimized in the course of the training, since the magnitudeof this value indicates the inaccuracy of the predictions of thepredictive model.

A “facility” for the production and testing of compositions as usedherein is an apparatus or system comprising several laboratory devicesand a transport unit, which is capable of jointly controlling thelaboratory devices and the transport unit in an orchestrated manner inorder to carry out a workflow automatically or semi-automatically. Theworkflow can be, for example, a coating composition preparation workflow(e.g. a combine-and-mix workflow), or an analysis workflow or acombination of two or more of these workflows. The workflow can compriseautomatically preparing a coating composition and/or automaticallyapplying the composition on one or more substrates and/or automaticallyobtaining and processing digital images of the coating surface of thesubstrate(s). The facility can be, for example, a high-throughputfacility (HT-facility), also referred to as “high-throughput equipment”(HTE).

The “testing” or “analysis” of a coating composition by an automatedmanufacturing and/or testing facility is the process of analyzingchemical, physical, mechanical, optical or other empirically measurableproperties of the composition by means of one or more analysis modules.For example, the testing may comprise applying a composition on asubstrate, acquiring and analyzing a digital image of the coatingsurface, and computing a quality measure of the coating as a function ofone or more defects detected in the digital image of the coatingsurface. The analysis may further comprise measuring further objectproperties, e.g. opaqueness, elasticity, rheological properties, color,etc.

An “active learning module” as used herein is a software program or amodule of a software program which is designed to select a(comparatively small) subset of candidate compositions from a set ofcandidate compositions in such a way that a particularly strong learningeffect occurs after the preparation and empirical measurement of theproperties of this selected candidate composition as a consequence ofthe consideration of these data in training the predictive model.

A “model” or “predictive model” as used herein is a data structure orexecutable software program or program module configured to generate aprediction based on input data. For example, the model can be a modelobtained in a machine-learning process by training the model on manuallyand/or automatically annotated training data. A predictive model as usedherein may also comprise a collection of two or more functionallyintegrated models, e.g. a set of models used and/or comprised in anapplication program. The predictive model can be, for example, a neuralnetwork model, a support vector model, a random forest, a decision tree,or the like. According to embodiments of the invention a predictivemodel adapted to compute a characterization of a coating surface inrespect to the presence, location and/or extent of one or more coatingdefects based on digital image data is also referred to as “Ml” model. Apredictive model adapted to predict a property of a coating composition,in particular a characterization of a coating surface generated based onthis composition, based on one or more input parameters related to e.g.the components, component amounts, manufacturing process parametersand/or application process parameters of this composition is alsoreferred to as “M2” model. A predictive model adapted to predict on oneor more parameters related to e.g. the components, component amounts,manufacturing process parameters and/or application process parametersof a coating composition based on input data specifying a desiredproperty of a coating composition, in particular a desiredcharacterization of a coating surface generated based on thiscomposition, is also referred to as “M3” model. According toembodiments, the model M3 is configured to take into account anincomplete coating composition specification as additional input data,whereby the incomplete coating composition may be used to limit thesolution space of the prediction to be provided by the model M3.

BRIEF DESCRIPTION OF THE DRAWINGS

In the following, only exemplary forms of the invention are explained inmore detail, whereby reference is made to the drawings in which they arecontained. They show:

FIG. 1 a flowchart of a method for automated characterization of acoating surface;

FIG. 2 a flowchart of a method for automated characterization of acoating surface in greater detail;

FIG. 3 a flowchart of a method for coating a surface and for obtainingan image of a coated surface;

FIG. 4 a block diagram of a data processing system for automated coatingsurface characterization;

FIG. 5A a data processing system in the form of a smart phone comprisinga locally installed app;

FIG. 5B a data processing system in the form of a smart phone comprisinga web-application;

FIG. 5C a data processing system in the form of a customized coatingsurface quality control device;

FIG. 5D a data processing system in the form of a computer coupled to afacility for manufacturing coating compositions;

FIG. 6 an illustration of the training and test phase for generating andusing the defect-identification program;

FIG. 7 an illustration of the training and test phase for generating andusing the composition-quality prediction program;

FIG. 8A a plurality of coating compositions;

FIG. 8B a “draw down” coating application unit;

FIG. 8C a “spraying” coating application unit;

FIG. 8D an automated conveyor belt for transporting coating samples in aHTE;

FIG. 9A a digital image of a coating surface before manual annotation ofthe defects;

FIG. 9B the digital image of FIG. 9A comprising manually labeled(annotated) defects;

FIG. 10 a digital image of a coating surface with automaticallyidentified and labeled defects;

FIG. 11 measures of a defect of a surface area in the form of ahistogram and an area percentage value;

FIG. 12 an architecture of a neural network used for predicting thequality of a coating composition; and

FIG. 13 an illustration of the relative positioning of a camera, a lightsource and the coated surface for acquiring a digital image;

FIG. 14 a flow-chart of a method for identifying candidate coatingcomposition best suited to improve the quality of a predictive model ofthe composition-quality-prediction program;

FIG. 15 a block diagram of a distributed data processing and coatingmanufacturing system used for improving the predictive model of thecomposition-quality-prediction program; and

FIG. 16 a 2D section of a multidimensional data room for variouscombinations of a defoamer DF1 with optimum anti-foam property anddefoamer DF2 with optimum compatibility in specific coating composition,from which the “Active Learning Module” selects specific data points.

DETAILED DESCRIPTION

FIG. 1 shows a flow chart of a method for automated characterization ofa coating surface. In a first step 102, a defect-identification programprocesses a digital image depicting a coating surface. Thedefect-identification program identifies one or more coating defects andprovides a characterization of the recognized effects in step 102. Forexample, the program may determine that the coating surface comprises afirst bubble defect region comprising about 100 bubbles and a secondbubble defect region comprising about 400 bubbles. The characterizationof the recognized effect may comprise the type, location and extent ofthe identified defects. The data obtained in step 104 may be output to auser and/or may be used internally by the defect-identification programfor computing derivative data values, e.g. aggregated coating surfacecharacterizations.

FIG. 2 shows a flowchart of a method for automated characterization of acoating surface in greater detail. After steps 102 and 104, thedefect-identification program in step 106 computes measures of theindividual defects, e.g. size, shape, spatial distribution, sizedistribution or the like. The defect-identification program uses thesemeasures in step 108 to compute and provide a qualitative and/orquantitative characterization of the coating surface.

FIG. 3 shows a flowchart of a method for coating a substrate and forobtaining an image of the substrate's coated surface. In step 110, aplurality of coating compositions is generated by mixing multiplecomponents with each other according to a mixing and manufacturingprotocol. In step 112, a coating composition is applied in accordancewith a coating application protocol on one or more substrate samples.The samples are transported automatically or manually to an imageacquisition unit. In steps 114 and 118, a camera, an optional lightsource and the coated sample are positioned relative to each other in adefined manner as to enable an image analysis software 124 to correctlyanalyze the images. Then, one or more images depicting the coated samplesurface are taken in step 118.

FIG. 4 shows a block diagram of a data processing system 120 forautomated coating surface characterization. The data processing systemcomprises one or more processors 126 and a volatile or nonvolatilestorage medium 122. The storage medium can comprise images 125, e.g.training images for training the model M1 of the defect-identificationprogram 124 or test images to be input to the already trained predictivemodel M1. In addition, or alternatively, the storage medium can comprisetraining data for training the model M2 of the composition-qualityprediction program and/or can comprise the composition-qualityprediction program comprising an already trained predictive model M2.

In addition, or alternatively, the storage medium can comprise anuntrained version of the model M3 of the composition-specificationprediction program and/or can comprise the composition-specificationprediction program comprising an already trained predictive model M3.

The data processing system 120 can be implemented in many differentways. For example, the data processing system can be a monolithiccomputer system, e.g. a desktop computer system, a portabletelecommunication device, a smart phone, a special purpose coatingsurface-quality control device or a computer system being operativelycoupled to or being an integral part of a facility for automaticallymanufacturing and/or testing coating compositions. Alternatively, thedata processing system 120 can be a distributed computer system, e.g. aclient/server computer system optionally coupled to one or morefacilities for automated manufacturing and/or testing of coatingcompositions. The components of the distributed computer system can becommunicatively linked with each other via a network connection, e.g.the Internet or an intranet of an organization. FIGS. 5A-5D illustratesome implementation examples of the data processing system 120.

FIG. 5A shows a data processing system in the form of a smart phone 130.The smart phone comprises the defect-identification program in the formof a locally installed application program, also referred to as “app”124.

For example, the app may comprise a graphical user interface 132 whichallows a user to control the camera 134 of the smart phone as to take animage of a coated sample surface at an appropriate distance and positionrelative to the coated surface. The successful acquisition of thedigital image may automatically trigger the defect identification app124 to analyze the image and to identify one or more coating defectsdepicted in the image. Preferably, the app 124 is configured to generatethe further GUI 136 informing a user on the result of the processing.For example, GUI 136 may indicate the type of identified defects (bubbledefects) and one or more quantitative measures of the defects (e.g.average bubble diameter, bubble density, etc.) In addition, the GUI 136comprises a characterization of the coating surface computed as afunction of the properties of the one or more identified coatingdefects. For example, the defect-identification program can beconfigured to compute, based on the sizes and number of the identifiedbubbles, the total size of the area of the surface covered with bubbles.

Using a smart phone app for acquiring images of coating surfaces and forautomatically identifying coating defects may have the advantage that itis not necessary to equip the employees of a company with extensivespecial-purpose devices in order to obtain an objective, reproduciblequality measure for coating surfaces. It is sufficient to download andinstall an app.

FIG. 5B shows a data processing system in the form of a smart phone 130comprising the defect-identification program 124 in the form of aweb-application.

According to one example, the defect-identification program isimplemented as a script that runs in the browser of the smartphone andthat is downloaded by a user visiting a particular website, e.g. a webportal of a company generated by a server 144 and offered via theInternet or intranet. For example, the program 124 can be implemented asJavaScript program.

According to another embodiment, the defect-identification program isimplemented as a program running outside of a browser, e.g. a Javaprogram.

The defect-identification program can be implemented as a two-componentprogram comprising a client portion and a server portion which areinteroperable and are configured to exchange data via a networkconnection 142. For example, the program portion installed on theportable telecommunication device 130 (“client application”) can beconfigured to controlling the image acquisition process and foroutputting the defect identification results to a user. The programportion installed on the server (“server application”) can be configuredto receive the digital image from the client portion via the network, toanalyze the digital images for detecting coating defects, fordetermining measures of the identified defects and for computing aqualitative and/or quantitative characterization of the depicted coatingsurface. The server portion returns the characterization and preferablyalso the measures and an indication of the type and extent of theidentified defects to the client portion.

FIG. 5C shows a data processing system 150 in the form of a customizedcoating surface quality control device, i.e., a dedicated hardwaredesigned for controlling and objectivizing the quality of coatingsurfaces and, implicitly, the quality of the coating composition and/orthe coating process. The device comprises a storage medium with thedefect-identification program 124, an interface 152 allowing a user tocontrol a testing process and output the test result, and preferablyseveral hardware components used for the purpose of testing the surfaceproperties of coated samples. For example, the device can comprise acamera 134 coupled to the device via a robotic arm 158 or via otherconnecting elements which allow to modify the relative position ofcamera and coated sample. The device can comprise one or more lightsources 160 which are preferably also coupled to the device via movableand/or rotatable connecting elements, e.g. a robotic arm 156.

The quality control device 150 can be implemented as portable device oras stationary device. For example, the device can be implemented as anintegral part of a facility for automatically manufacturing and/ortesting coating compositions. The facility comprises a conveyor belt 154for transporting a plurality of coated samples 162, 164, 166, 168 to thecontrol device 150, thereby enabling a fully automated, fast andreproducible quality control of many coating surfaces.

FIG. 5D shows a data processing system in the form of a computer 170coupled to a facility 244 for manufacturing coating compositions.

The facility 244 comprises a main control computer 246 for controlling,monitoring and/or orchestrating various tasks related to themanufacturing of coating compositions, related to the application ofcoating compositions on various surfaces and/or related to the testingof the coated surfaces or of the coating compositions(e.g. fordetermining rheological, chemical, physical or other parameters of thecoating composition). The respective tasks are performed by severaldifferent units comprised by the facility 244. For example, the facilitycan comprise one or more analyzers 257 for performing chemical,physical, mechanical, optical or other forms of tests and analysis oncoating compositions or on substrate surfaces having been coated with acoating composition. The facility can comprise one or more mixing units256 configured for manufacturing various coating compositions, e.g. bymixing the components of a composition based on a specific manufacturingand mixing protocol. The facility can comprise one or more samplecoating units 254 configured for automatically coating surface samples.For example, the coating units 254 can comprise a “spraying” coatingapplication unit or a draw down” coating application unit as depicted,for example, in FIGS. 8C and 8B. According to some exampleimplementations, the facility further comprises an image acquisitionunit 252 comprising a camera and a light source and means forpositioning the sample and the camera and/or light source relative toeach other such that the acquired digital images can be used as input bythe defect-identification program 124. One or more transport units 258,e.g. conveyor belts, connect the different units and carry components,mixtures, coating compositions and coated samples from one unit to theother.

The control computer 246 comprises a control unit 248 configured forsending the digital images of coated samples acquired in the imageacquisition unit 252 to the defect-identification program 124 of thecomputer system 170. According to some embodiments, additionalparameters are provided together with the image data to thedefect-identification program. The parameters may indicate the identity,relative and/or absolute amounts of one or more components of thecoating composition used for coating a sample, and optionally alsomanufacturing process parameters and/or application process parameters.The parameters can be provided in the form of a complete or incompletespecification of the coating composition and/or the respectivemanufacturing or application process parameters.

The defect-identification program is configured to use the receivedimage, and optionally also the parameters, as input for automaticallyidentifying coating defects depicted in the image, for computing defectmeasures and for computing coating surface characterizations thefunction of the defect measures. The results computed by thedefect-identification program can be output to a user via a GUI and/orcan be stored in the database 204.

Preferably, parts of the data obtained by the other units such as theanalyzers 257 or the mixing units 256 or the coating units 254 can bestored directly in the database in association with an identifier of aparticular coating composition and/or with an identifier of coatedsamples or can be sent to the computer system 170 to have the computersystem 170 store the data in the database.

Using the defect-identification program in the context of the facility244 can be particularly advantageous, because after the images of thecoating surfaces have been taken, they can be automatically analyzed forthe defects to be examined. The result obtained can be linked to theformulation data and/or analysis data and thus be used to optimize thecomposition.

FIG. 6 shows an illustration of the training and test phase forgenerating and using the defect-identification program.

In a first step, a training data set 602 is generated. For this purpose,a plurality of different coating compositions is prepared. The differentcoating compositions vary in respect to the nature of their components,the amounts of the respective components, and/or in respect tomanufacturing process parameters. The multiple coating compositions arethen applied on substrates in order to generate a plurality of coatingsurfaces. The number of coated samples can be much larger than thenumber of coating compositions, because the same coating composition canbe applied on many different types of material (wood, plastic,cardboard, metal etc.), via many different types of coating techniques(spraying, painting, immersing, spreading, etc.). Then, one or moredigital images 604, 606 are acquired for each of the coated samples. Forexample, for a particular coating sample, digital images can be obtainedby varying illumination strengths, the relative position of the lightsources, the light wavelength, and the like.

Depending on the coating composition, the material of the sample, thecoating process parameters and many other factors, the coating surfacesdepicted in the images may comprise one or more coating defects ofdifferent predefined defect types.

In the next step, the defects depicted in the acquired digital imagesare labeled manually. For example, the labeling process can be performedas described with reference to FIG. 9B.

The digital image 604 can be annotated with labels 616 being indicativeof the location, type and preferably also the extent of each coatingdefect depicted in the image 604. The labeled 616 also comprises aqualitative and/or quantitative characterization of the image 604, e.g.“overall quality level 7 comprising bubble defects of grade 2 andwrinkling defects of grade 8”. Preferably, the image 604 is stored inassociation with additional data 608. The additional data 608 cancomprise a complete or incomplete specification of the components of thecoating composition used for generating the depicted coating surface,whereby the specification may in addition comprise component namesand/or amounts, manufacturing process parameters and/or compositionapplication process parameters. For example, a specification of acomposition and the above-mentioned parameters can be stored in adatabase in association with an identifier of the coating compositionused for generating the depicted coating surface. The image 604 can havean image ID stored in association with an identifier of the coatedsample depicted in the image, whereby the identifier of the coatedsample stored in association with an identifier of the coatingcomposition used for coating the sample.

The digital image 606 can be annotated with labels 618 and stored inassociation with additional data 610 analogously.

A computer system 120.1 is provided comprising an untrained version of apredictive model 612 that is to be trained. The computer system hasaccess to the database 204 comprising the training data. The trainingdata 602 is used as input during the training process. In the trainingprocess, the predictive model M1 learns correlations between pixelpatterns in the annotated images 604, 606 and coating defects/coatingsurface characterizations.

According to embodiments, the training data comprises additional data608, 610 in the form of parameters. The predictive model M1 (or furtherpredictive models M1.2 comprised in the defect-identification program)will also learn correlations between pixel patterns, coatingdefects/coating surface characterizations and the additional data suchas the components and/or amounts of the composition components,manufacturing process parameters and/or application process parameters.

The untrained model can be implemented in the form of a neural network.The neural network preferably comprises a region proposal network, e.g.a region proposal network provided by the Mask R-CNN program.

As a result of the training, a predictive model M1 (which may compriseone or more further predictive models M1.2) 612 is provided havinglearned the above-mentioned correlations. The trained model can beintegrated in a defect-identification program 124 and used forautomatically identifying coating defects depicted in digital images.The program 124 may comprise additional functionalities, e.g. a GUI 614for assisting a user in acquiring images during training and/or testphase and/or for displaying the prediction result to a user, e.g. in theform of numerical values and/or segmented images.

The lower part of FIG. 6 shows the test phase of the already trainedmodel Ml. During the test phase, a plurality of digital images ofcoating surfaces referred to as “test images” are provided. For example,the test images 624, 626 can be stored in the same database 620 or in adifferent database used for storing the training images. The test imagesdo not comprise any label at first. Optionally, the test images can bestored in association with additional data 628, 630, in particular acomplete or incomplete specification of the nature and/or amounts of thecomponents of the coating compositions, various manufacturing protocolparameters, coating composition application process parameters and/orimage acquisition system parameters.

In addition, a computer system 120.2 is provided on which a copy of thedefect-identification program 124 is installed and/or instantiated. Thecomputer system 120.1 used for training the model can be the samecomputer system 120.2 used for applying the trained model on test imagesor can be a different computer system to which a copy of thedefect-identification program was provided.

The defect-identification program 124 comprising the trained predictivemodel M1 612 receives one or more test images 624, 626 which are used asinput for predicting a qualitative and/or quantitative characterization632, 634 of the coating surface depicted in the respective test image.For example, the defect-identification program can perform a pixelwiseimage analysis for identifying the location, type and extent of coatingdefects depicted in the image. Then, the program can analyze theobtained data for computing measures of each coating defect identifiedin an image. For example, the fraction of pixels representing bubbledefects and the fraction of pixels of the same image representing adelamination defect can be determined. In a further step, these measuresare used for computing an aggregate characterization of the coatingsurface, e.g. a label such as “bubble defect grade 3” as a function ofthe defect measures.

Optionally, the additional data 628, 630 assigned to the respective testimage is received and used by the defect-identification program asadditional input for performing the prediction.

Preferably, the automatically predicted measures and characterizations632, 634 are stored in association with the respective test images 628,613 in the database.

Thereby, data is provided comprising labels which are indicative ofcoating surface characterizations and hence, the coating quality of aparticular coating composition or protocol. The data can be used forincreasing the training data for model M1 602 and for retraining thepredictive model based on the extended database to improve accuracy. Inaddition, or alternatively, the data can also be used for traininganother predictive model M2 with a different scope of prediction asexplained with reference to FIG. 7.

FIG. 7 shows an illustration of the training and test phase forgenerating and using a composition-quality prediction program 714.Contrary to the defect-identification program, the composition-qualityprediction program 714 does not require a digital image as input.Rather, the predictive model M2 712 used by the composition-qualityprediction program uses a plurality of parameters as input. These inputparameters are selected from a group comprising components, componentamounts, manufacturing process parameters and/or application processparameters. The quality related characterizations of a coating surfaceand optionally further coating composition quality indicators arecomputed and output by the coating composition quality predictionprogram.

The combined use of a defect-identification program and acomposition-quality prediction program may be beneficial, because forthe first time, the defect-identification program provides qualitycharacterizations of a coating surface in sufficient amount, quality andobjectivity as to allow the use of this data for training a differentmachine learning programs 712 to solve a different task, e.g. qualityprediction of a coating composition, whereby the quality of a coatingsurface is also taken into account.

During the training phase of the model M2, training data 702 isprovided. The training data 702 comprises a plurality of data records(here: two data records represented as circles), whereby each datarecord represents a coating composition. Each data record can comprise acomplete or incomplete specification 628, 630 of the nature and/oramount of the components of the composition, a specification ofmanufacturing process parameters and/or of coating compositionapplication process parameters. In addition, each data record comprisescharacterizations 632, 634 of a coating surface generated by a) coatinga sample with the respective coating composition, b) acquiring an imageof the coated surface, and c) analyzing the image by thedefect-identification program for computing the coating surfacecharacterization 632, 634. In addition, each data record can compriseone or more further properties 732, 734, in particular rheologicalproperties, shelf life, density, etc.

During the training phase, the machine learning model 712 learns tocorrelate the coating surface characterizations 632, 634, the datacomprised in the specifications 628, 630, and the additional properties732, 734, if any.

As a result of the training, the trained predictive model M2 is able topredict the quality of the coating composition 732, 734 as a function ofits components and associated process parameters 728, 730, if any. Forexample, the quality of the coating composition can be predicted andprovided in the form of quality characterizations 732, 734 of a coatingsurfaces as output by the defect-identification program. In otherembodiments, the quality of the coating composition can be predicted andprovided in the form of a combination of the quality characterization ofthe coating surface and other property values 729, 731, e.g. anindication of the shelf life, viscosity and the like.

The training of the predictive model M3 to be used by acoating-specification-prediction program can be performed analogously asdescribed for model M2, whereby the same correlations are learned butthe input data of M2 is learned as the output data of M3 and the outputdata of M2 is learned to be used as input data of M3. Optionally M3 usesadditional data like an incomplete coating specification provided tolimit the solution space that has to be evaluated by model M3.

FIG. 8A shows a plurality of coating compositions having beenmanufactured automatically in an HTE based on a respective coatingcomposition specification.

FIG. 8B shows a “draw down” coating application unit. In a first step, acertain amount of the coating composition is applied to the surface ofthe sample. A roller is then moved over the surface at a defineddistance from the surface, whereby the amount is spread evenly over thesurface and form the coating.

FIG. 8C shows a “spraying”-coating application unit.

FIG. 8D shows an automated conveyor belt of a high-throughput facilityfor manufacturing and/or testing coating compositions. The conveyor beltis adapted to automatically transport the coated samples to an imageacquisition unit of the facility. The samples are loaded into the imageacquisition unit and the camera, light source and/or the sample aremoved and positioned relative to each other such that the imagesacquired from the surface of the sample are suited for automatedprocessing by the defect-identification program.

FIG. 9A shows a sub-area of a digital image 1202 of a coating surfacebefore manual annotation (label assignment) of the defects. For example,the image may have been acquired by a camera in an image acquisitionunit of a facility for automatically manufacturing and/or testingcoating compositions. Alternatively, the image may have been obtained bya camera of a data processing system described for example withreference to FIGS. 4 to 5.

The image acquisition conditions (lighting, camera settings, imageacquisition angle, illumination angle, etc.) are selected so that thedefect to be examined is shown as well as possible on the images.According to embodiments, different image acquisition conditions are setand used for different types of defects. For example, defects associatedwith elevations or depressions of the surface may be analyzed based onimages obtained using a shallow illumination angle (i.e., a shallowlight incidence angle) in order to ensure that the defects cause shadowsof sufficient size and contrast. Other defects, e.g. color defects, maybe analyzed based on digital images obtained using a steep illuminationangle (of about 80-100°).

The digital image 1202 shows a coating surface containing foam defects.The coating surface comprises two major defects (holes) 1204, 1206, andseveral smaller defects (small holes). As the substrate was illuminatedfrom the side, the defects are clearly recognizable due to the shadowformation. The shadow formation allows the identification anddifferentiation of elevations and depressions of the coating on thesubstrate. In addition, the shadow formation can be used to judgewhether, for example, in the case of a depression, there is a sharpedge, or the coating thickness is slowly decreasing. This may allowdiscerning bubble defects from cratering defects.

In addition to the defects, the digital image comprises an artifact 1208which is not a coating defect. For example, the artifact can be causedby a dust speckle on one of the lenses of the image acquisition systemor on the substrate.

In order to generate a training data set of sufficient size, digitalimages of many different coating surfaces comprising many differenttypes of coating defects are acquired. For the foam defects, a shallowlight incidence angle is chosen. For other types of defects, other imageacquisition settings and conditions may be chosen. Preferably, thesurface of each coated sample is illuminated under many differentconditions and a respective digital image is acquired in order to beable to generate a defect-identification program that is able toidentify many different types of defects which may overlay each other.

Preferably, a large number of digital images showing several thousanddefects of different defect types and different types of coatedsubstrates is acquired which are manually annotated (labelled).

FIG. 9B shows the digital image 1202 of FIG. 9A comprising manuallylabeled (annotated) defects. Annotating a digital image may comprisestoring the digital image in association with information beingindicative of the position and type of the coating defects depicted inthe image. Optionally, the labels may comprise additional data such asimage resolution which may allow determining quantitative defectmeasures such as diameter, circumference, or the like.

Annotating many images manually consumes a considerable amount of timeand effort. In order to ease matters, images where cut into smallersections. For some of these sections, the foam defects were marked withthe software VIA -VGG Image Annotator (Abhishek Dutta and AndrewZisserman, 2019, “The VIA annotation software for images, audio andvideo”, Proceedings of the 27th ACM International Conference onMultimedia (MM '19), Oct. 21-25, 2019, Nice, France. ACM, New York,N.Y., USA, 4 pages. https://doi.org/10.1145/3343031.3350535.).

The defects 1206 and 1208 have been manually marked by manually drawncircles 1212 around each defect using the VIA software. The VIA softwarewas then used to export the marking information in a structured formatin association with the image or an image identifier. The structuredformat can be, for example, an XML file, a JSON file, a comma separatedfile, data records in a relational database or the like.

According to some embodiments, additional data is stored in associationwith the image or the image identifier. The additional data can comprisea complete or incomplete specification of the coating composition usedfor creating the coating surface, whereby the specification may comprisean indication of the identity and/or amounts of the components,manufacturing process parameters being indicative of aspects of theprocess of manufacturing the composition, application process parametersspecifying aspects of the process of applying the composition on thesubstrate and/or image acquisition system parameters. This may allow thepredictive model M1 (or additional predictive modes M1.2 used by thecoating quality prediction program) to learn correlations between defecttypes and coating surface characterizations on the one hand and one ormore of the above-mentioned parameters.

The generation of the predictive model M1 to be used by thedefect-identification program can be performed, for example, asdescribed with reference to FIG. 6.

The trained model M1 can then be used to detect foam defects (and othertypes of defects covered by the training data) in new, unlabeled imageswhich have not been used in the training step (and which are referred toas “test images”).

FIG. 10 shows a digital image 1302 of a coating surface. The imagecomprises labels 1304 which have been automatically created by adefect-identification program 124 according to an embodiment of theinvention. The labels identify defects 1306, 1308 having beenautomatically detected by the defect-identification program.

In addition to the visual representation of the detected defects (e.g.via image segments overlaying identified defects or, as in this case,via edges and circles surrounding the identified defects), thedefect-identification program is configured to temporarily orpermanently store the types and locations of the identified defect alsoin a structured form. For example, the location can be stored in theform of pixel coordinates of pixels representing a defect. Storing theidentity and location of the defects in structured form allows thedefect-identification program to process the structured data to computeaggregated characterizations of the coated surface.

For example, the aggregated characterization of the whole surface may bea quantitative characterization of the coating surface, e.g. a scalevalue referring to a quality scale with more than 5, e.g. more than 10possible scale values, whereby the scale may represent the integratedcoating surface quality. The quality score of a surface negativelycorrelates with the size and number of defects identified in the coatingsurface.

Another example for a quantitative defect measure and/or for aquantitative characterization of a coating surface would be a histogramof defects of different sizes as depicted in FIG. 11.

FIG. 11 shows measures of a defect of a surface area in the form of ahistogram 1402 and an area percentage value. The histogram and/or thepercentage value may be used as a quantitative characterization of thecoating surface or may be used to computationally derive such acharacterization.

The histogram 1402 depicts the distribution of foam bubbles of differentsizes, wherein a bubble size is specified as the area of a bubblemeasured in pixels. The bubble sizes are grouped into 10 different binsand the number of bubbles having a size falling in the size range of abin are plotted. Hence, the histogram provides a rough estimate of thesize distribution of the bubbles which may allow identifying problems inthe coating composition or the coating process.

In addition, the fraction of the surface covered by bubble defects iscounted (5.88% in this case). This value can be used as qualitycharacterization of the coating surface depicted in the processeddigital image. By evaluating the number and size of the defects, it ispossible to classify a coating surface into predefined quality classesor grades in an objective, reproducible manner.

FIG. 12 shows an architecture of a neural network 400 encoding apredictive model M2 and being used for predicting the quality of acoating composition.

The network 400 is configured and trained to receive an input vector 402and to calculate and output an output vector 406 as a function of theinput vector.

For example, the input vector can encode a complete or incompletespecification of the components (and optionally also the concentrationsor quantities of the respective components) of a coating composition.Optionally, the input vector may further comprise process parametervalues of a coating composition manufacturing process and/or of acoating application process.

The output vector 406 specifies one or more properties of thecomposition or of a surface generated by coating a substrate with thecomposition. The properties preferably include one or more propertiesrepresenting the quality of the coating composition and/or the qualityof a coating surface formed by the coating composition.

The network includes several layers 404 of neurons, which are linkedwith the neurons of other layers by means of weighted mathematicalfunctions in such a way that the network can calculate, i.e. predict,the properties and quality characterizations of the correspondingcompositions and of coating surfaces generated from the saidcompositions on the basis of the information encoded in the input vectorand can output the predicted properties and quality characterizations inthe form of an output vector 406.

Before training, the neurons of the neural network are first initializedwith predetermined or random weights. During training, the networkreceives a specification of a coating composition (which may comprisethe type and optionally also the amounts of the components andoptionally manufacturing process parameters or application processparameters) together with empirically measured properties of thiscomposition, including defect measures and coating surfacecharacterizations computed and output by a defect-identification program124. The network calculates the output vector with predicted propertiesand quality measures of this composition and is penalized by the lossfunction for deviations of the predicted properties and quality measuresfrom the known, empirically determined properties and coating surfacecharacterizations. The determined prediction error is distributed backto the respective neurons that caused it via a process calledbackpropagation and causes the weights of certain neurons to change insuch a way that the prediction error (and thus the value of the lossfunction) is reduced. Mathematically, this can be done by determiningthe slope of the loss function, so that the neuron weights can bedirectionally modified to minimize the value output by the lossfunction. Once the prediction error or loss function value is below apredefined threshold, the trained neural network is consideredsufficiently precise so that further training is not necessary.

After the training is successfully completed, the trained predictivemodel M2 can be used to predict the properties, in particular qualitycharacterizations of a coating surface of a new, unknown coatingcomposition. In case the exact composition and/or the optimummanufacturing process parameters and/or coating process parameters arenot known, a human user or an accessory software program generateseveral candidate coating composition specifications which represent andspecify variants of the new coating composition of interest. The trainedneural network is used to automatically predict the properties of eachof the new (candidate) coating compositions. In case the prediction wasperformed for multiple candidate coating compositions, the one or moreof the candidate coating compositions having the best properties orquality characterizations for the respective application scenario areselected and/or are actually produced and tested in the facility.

The predicted properties of each candidate coating composition areoutput as output vector 406 of the neural network to a user for manualevaluation and/or are stored in a database, e.g. for further evaluationand comparison with empirically obtained property values of thecomposition which may be obtained later. The input vector can contain,for example, 20 components of a coating composition, some manufacturingprocess parameters and some application process parameters. The outputvector 406 may comprise various properties whose nature depends on thetraining data used for training the predictive model M2 encoded in thenetwork. For example, the output vector can comprise an indication ofthe type and extent of coating defects which will likely occur if thecoating is applied on a substrate.

FIG. 13 shows an illustration of the relative positioning of a camera134, a light source 160 and the coated surface 164 for acquiring adigital image. For example, the relative positioning can be performedmanually, e.g. in case the camera is a smartphone camera. In otherembodiments, the relative positioning can be performed automatically,e.g. within an image acquisition unit of a facility 244 formanufacturing and/or testing coating compositions. By performing therelative positioning, it is ensured that the image acquisition angle1504 and the illumination angle 1502 are chosen such that the digitalimage of the coting surface will allow an automated identification ofthe coating defects by the defect-identification program 124. Forexample, the angles 1502, 1504 should typically be identical or similarto the angles used for obtaining the digital images used for trainingthe predictive model M1 of the defect-identification program. Forexample, an angle “similar” to angle X can be an angle in the range of Xup to plus or minus 40% of X, in particular X up to plus or minus 20% ofX. according to embodiments, multiple digital images are obtained foreach coated sample, whereby the image acquisition angle 1504 and/or theillumination angle 1502 differ from each other. This may ensure that formany different types of defects, the set of acquired images comprisesone or more images which allow accurate identification andcharacterization of a defect.

FIG. 14 shows a flow-chart of a method for identifying a coatingcomposition best suited to improve the quality of a predictive model M2(or M3) of the composition-quality-prediction program.

The process can, for example, be carried out by a computer system 120.2as shown in FIG. 7 or computer system 170 depicted in FIG. 5D.

In a first step 802 a), already known compositions and their propertiesand quality characterizations, including characterizations of thecoating surface generated by the composition, are used as an “initialtraining data set” to train a model M2 (or M3), e.g. a neural network, asupport vector machine, a decision tree, a random forest or the like.The trained model M2/M3 can be integrated in a composition-qualityprediction program configured to predict properties of a coatingcomposition, e.g. the quality of a coating surface created by thecoating composition, or into a composition-specification predictionprogram

In the next step 804 (b), a check is performed to determine whether thevalue of a loss function meets a predefined criterion. A fulfillment ofthe criterion expresses that the prediction accuracy of the trainedneural network is considered sufficient. Selectively for the case thatthe criterion is not fulfilled, the steps 806-812 described below areperformed. Otherwise the training is terminated (step 814) and thetrained neural network is returned.

In step 806, the Active Learning module automatically selects aspecification of a candidate coating composition from a plurality ofmanually provided or automatically computed candidate coatingcomposition specifications. There exist several different ActiveLearning approaches that can be used according to embodiments of theinvention.

According to one embodiment, the Active Learning module follows the“expected model change” approach and selects a specification of thecandidate coating composition that (when the network is re-trainedtaking into account this candidate coating composition and its realmeasured properties) would change the current predictive model the most.

According to another implementation variant, the Active Learning Modulefollows the “expected error reduction” approach and selects thecandidate composition that would most strongly reduce an error of thecurrent predictive model of the trained neural network.

According to another implementation variant, the Active Learning Modulefollows the “Minimum Marginal Hyperplane” approach and selects theexperimental composition that is closest to a dividing line or planethat is spanned in a multidimensional data space by the currentpredictive model of the trained model. The dividing line or dividingplane are interfaces within the multidimensional data space in which thepredictive model makes a classification decision, i.e. assigns datapoints on one side of the dividing line or dividing plane to a differentclass or category than the data points on the other side of the dividingline. This proximity of the data points to the parting plane isinterpreted in such a way that the predictive model is uncertain about aclassification decision and would benefit to a particularly high degreeif real measured data sets (consisting of a combination of componentsand optionally their concentrations and the measured properties of thecoating composition produced according to this composition of thecomponents) from the vicinity of this parting plane were additionallymeasured in order to further train the model.

After retrieving the specification of the selected candidate coatingcompositions from the database, in step 808 the computer system controlsa composition production and testing facility 244 so that a product isautomatically produced and tested according to the retrievedspecification. This testing is understood to be a metrological recordingof one or more properties of the product, e.g. the measurement of pHvalue, color value, viscosity, the computation 809 of a characterizationof a coating surface created by coating a sample with the composition,taking an image and having the image analyzed by a defect-identificationprogram, or the like.

The real measured properties obtained in step 108 and the computedvalues derived from image data obtained in step 809 are used tosupplement the selected candidate coating composition, so that acomplete further data point consisting of a known composition and knownproperties is obtained, which serves to extend the training data setused in a) of the current or previous iterations.

In step 810, the model M2/M3 is retrained on the extended training dataset. Depending on the implementation variant, this can be done in such away that the training is carried out again completely on the basis ofthe extended training data set, or the training in step 810 isincremental, so that what has been learned so far is retained and onlymodified by taking the new training data point into account.

In step 812, a repeated check of the prediction quality of the trainedmodel M2/M3 is initiated and steps 804-812 are repeated until the modelhas sufficient prediction quality, which is indicated by the fact thatthe loss function fulfils the criterion, e.g. the “error value”calculated by the loss function is below a predefined maximum value. Thefully trained model can now be used to predict the properties of acoating composition, including quality characterizations of a coatingsurface obtained from this coating composition, very quickly andreliably. To do this, the re-trained model M2 is integrated in a qualityprediction program, whereby an older, less accurate version of the modelmay be replaced.

Since the model has learned the statistical correlations between variousparameters related to coating compositions (in particular theircomponents, absolute or relative amounts, manufacturing processparameters and/or application process parameters) and the properties ofthe resulting product (including characterizations of a coating surfacecreated from the composition), the trained model M2 can now predict theproperties and also the quality of a corresponding coating surface givena specification of a coating composition even for compositions for whichno empirical data is available. Likewise, a trained model M3 can predictone or more of the above-mentioned parameters related to a coatingcomposition given a desired coating surface characterization andoptionally an incomplete coating composition. Both the model M2 and themodel M3 rely on learned associations of empirically measurable orderivable coating composition properties such as a coating surfacecharacterization on the one hand and the above-mentioned variousparameters related to the coating composition. The models M2 and M3differ only in respect which of the said two aspects is expected asinput and which is provided as output.

A “loss function” (also called “objective function”) for a predictionproblem, can, for example, in the simplest case, only count thecorrectly recognized predictions from a set of predictions. The higherthe proportion of correct predictions, the higher is the quality of thepredictive model (e.g. a model implemented in a neural network) used ina machine learning process. For example, the question whether arheological property such as viscosity and/or a quality property such asthe type and extent of coating defects in a coating surface is within apredefined acceptable range can be understood as a classificationproblem.

However, many alternative loss functions and corresponding criteria forassessing the prediction accuracy of the trained model are alsopossible.

FIG. 15 shows a block diagram of a distributed data processing andcoating manufacturing system 900 used for improving the predictive modelof the composition-quality-prediction program;

The system includes a database 904 with known coating compositions 906and candidate compositions 908. The known compositions 906 may, forexample, be a set of data records each containing a complete orincomplete specification of the type and/or the amount of components, ofmanufacturing process parameters and/or coating composition applicationprocess parameters. In addition, each data record of the known coatingcompositions comprises empirically determined physical, chemical,haptic, optical and/or other metrologically ascertainable properties ofthe coating composition and/or a coating surface generated therefrom,whereby “empirically determined” includes measures and characterizationscomputed as a function of empirical data, e.g. quality characterizationcomputed as a function of image data.

The candidate compositions 908, on the other hand, are compositionswhose physical, chemical, haptic, optical and/or other metrologicallyascertainable properties are not known.

For example, the known compositions 206 can comprise coating compositionspecifications having already been produced and tested by an HTEfacility.

The coating compositions specifications 908 can comprise coatingcomposition specifications provided by a buyer of a coating component orcan be provided in a computational step of automatically creatingvariant coating composition specifications based on a single providedcomplete or incomplete coating specification.

For example, the specifications of the coating composition variants canbe created by increasing and/or decreasing the amount of one or morecomponents of this composition by 10%. If only one single component isvaried at a time, using a quantity of this component increased by 10%and a quantity reduced by 10%, two variants are formed per component.With 20 components, 40 candidate compositions are created with thisprocedure. The number of automatically generated candidate compositionsis preferably further increased by simultaneously increasing ordecreasing the concentration of two or more components compared to theirconcentration in the known composition and 10% and by modifying processparameters.

Typically, the number of automatically computed candidate compositionsis considerably larger than the number of coating compositions that alaboratory can actually physically prepare and test in terms of cost andprofitability.

The distributed system 900 includes a computer system 924, whichcomprises a neural network or another type of machine-learning model andan active learning module 922. The Active Learning Module 922 has atleast a read access to read one or more selected candidate compositionsand a specification of respectively assigned parameters, e.g. theircomponents, from the database 904. According to some embodiments, theActive Learning Module and/or a facility 944 which prepares and analyzesa coating composition and optionally also a coating surface createdtherefrom according to the selected candidate composition specification,also has write access to the database 904 in order to store theempirically obtained properties of the selected candidate coatingcomposition or of the respective coating surface in the database 904.For example, obtaining and storing coating surface characterizationsand/or measures of coating defects of a selected and newly preparedcandidate composition may result in this candidate composition becominga known composition and accordingly being stored in a different locationand/or provided with different metadata (“flag”) in the 904 database.

FIG. 16 shows a 2D section 1000 of a multidimensional data room forvarious combinations of a defoamer DF1 with optimum anti-foam propertyand a defoamer DF2 with optimum compatibility in a specific coatingcomposition, from which the “Active Learning Module” selects specificdata points 1008 in order to extend the training data set and to improvethe accuracy of the predictive model M3 used by thecomposition-specification prediction program. The model improvement isdescribed in the following for model M3, but can likewise be used forimproving the model M2 used by the coating composition qualityprediction program.

In the course of the training of the predictive model M3, which can beimplemented as a neural network or other type of machine learning model,the predictive model learns to calculate an output vector, which mayinclude one or more of the following: type of the components, absoluteand/or relative amounts of the components, coating compositionmanufacturing parameters, and/or coating composition application processparameters. The input data is preferably provided as input vector andcomprises one or more desired surface characterizations, e.g. qualitycharacterizations provided as input. Optionally, the input may comprisean incomplete specification of one or more components, absolute orrelative component amounts, manufacturing process parameters and/orapplication process parameters or respective amount or parametervalidity ranges. The incomplete specification can be used by the modelto limit the solution space to predicted coating compositions. Forexample, if the incomplete specification indicates a water-based coatingmedium, the predicted composition proposed by the model will not bebased on an organic coating medium. If the incomplete specificationindicates that two defoamers DF1 and DF2 should be used, only therelative amounts of these defoamers may be predicted by the model. Ifonly one defoamer DF1 is provided as input, the model M3 may onlysuggest a further defoamer DF2 which is predicted to be compatible withdefoamer DF1.

The input vector of the model M3 may include in particular one or moreof the following properties: coating surface quality measures, coatingdefect types (in particular bubble defects and cratering defects),coating defect measures, storage stability, pH value, rheologicalparameters, in particular viscosity, density, relative mass,coloristics, in particular color strength, and/or cost reduction duringproduction. The cost reduction during production can, for example, beautomatically recorded by an automated production facility 244 duringthe preparation of a composition and can, for example, relate to a givenreference value. However, it is also possible for a human to manuallyrecord the costs. Likewise, the coating defects and surface qualitymeasures can be automatically determined by the facility 244.

In the depicted example, the input received by the model may indicatethat both the occurrence of bubble defects and the occurrence ofcratering effects should be minimized. The input may further specify oneor more components of the coating composition, and in particular mayspecify that a defoamer DF1 with optimum anti-foam property and adefoamer DF2 with optimum coating medium compatibility in the(incompletely) specified coating composition should be used ascomponents, but no absolute or relative amounts of the defoamers isprovided. For example, the DF1 can be Evonik's Tego Foamex 810 and DF2can be Evonik's Tego Wet 285.

Based on the available data, the M3 model has learned that a mixture ofthe DF1 and DF2 defoamers produces a foam volume that lies between thetwo foam volumes produced by the two defoamers individually. The foamvolume-defoamer-ratio relationship almost represents a straight line,whereby the foam volume and the associated bubble defects decrease withan increasing ratio DF1:DF2, and thus the surface quality with respectto the bubble defects increases. Furthermore, the model has learned thatthe compatibility of the mixture of the two defoamers with the coatingmedium decreases with an increasing ratio DF1:DF2, i.e. the surfacequality with regard to the cratering defects decreases.

After some initial training steps, the model M3 of the neural networkhas already “learned” certain relationships between components orprocess parameters of the coating compositions and some properties ofthe resulting coating composition or coating surface. Based on theknowledge stored in the model M3, the model can predict that if theDF1:DF2 ratio in the incompletely specified composition provided asinput is too high, many cratering effects will occur, and that if theDF1:DF2 ratio is too low, many bubble defects will occur.

These learned relationships are illustrated in FIG. 16 by the dividingoutline 1016, which divides the data room with respect to the property“coating quality” into a data room 118 with acceptable coating surfacequality properties on the one hand and a data room 1020 withunacceptable coating surface quality properties on the other hand. Theplot illustrates that both the absolute and the relative amounts of thedefoamers may be of relevance: if the total amounts of both defoamers istoo low, bubble defects may occur. If the ratio DF1:DF2 is too high,cratering defects will occur. If the ratio is too low, bubble defectswill occur. If the total amount of both defoamers is too high, othersurface defects may occur or the production costs (not shown) may bepredicted to be unacceptably high.

FIG. 16 can only represent a partial aspect of the data room as a plotcan only represent two dimensions (“Concentration DF1” and“Concentration DF2”) but other coating components and their absoluteand/or relative amounts also have an impact on the coating surfacequality. A coating composition has often more than 10, typically about20 components, and as manufacturing process parameters and coatingapplication process parameters may also represent a respectivedimension, the data space 1000 comprises much more dimensions than shownin FIG. 16. Each of the many sub-spaces formed by these 20 dimensionscontains its own separating lines and areas of acceptable orunacceptable coating surface quality. The totality of separating lines1016 in the multi-dimensional space learned by the model M3 during thetraining is also referred to as separating plane (“hyperplane”).

The data points shown as circles in FIG. 16 each represent a candidatecoating composition for which the respective properties have not yetbeen determined empirically. The model M3 may be quite sure that thecoating compositions represented by e.g. data points 1008, 1009 liewithin an area of acceptable coating quality and that the coatingcompositions represented by data points 1004, 1012, 1010 haveunacceptable coating quality. However, the model may not be sure aboutthe surface quality of a coating composition represented by the datapoint 1002. So it can be assumed that preparing a coating compositionrepresented by data point 1002, empirically determining variousproperties of the coating composition (including coating surfacedefects), and using the obtained empirical data for extending thetraining data set and retraining the model M3 on the extended trainingdata set may provide the highest learning effect for the model.

For example, the data point 1002 to be used for extending the trainingdata with further empirical can be selected e.g. according to theso-called “Minimum Marginal Hyperplane” approach. For example, theActive Learning Module can be designed as a support vector machine or asanother algorithm capable of dividing a data space spanned by thetotality of data points into subspaces with respect to one or moreproperties based on the knowledge the predictive model has alreadylearned from a sub-set of the data points. Hence, the current“knowledge” already acquired during an initial training of the model M3is represented by the dividing line or dividing plane 1016. The “MinimumMarginal Hyperplane” method assumes that the data points 1002 with thesmallest distance to the dividing line 1016 are those for which thealready learned predictive model, represented by this dividing line1016, is most uncertain and therefore the experimental compositionbelonging to this data point should be selected, prepared and analyzedin order to empirically determine the actual properties, e.g. thecoating surface characterization and/or the viscosity. In the examplepresented here, the active learning module would thus select thecandidate coating composition represented by data point 1002, takinginto account only the property “coating surface quality”, and cause theplant 244 to prepare and analyze this composition in order to extend thetraining data with the specification of the components of composition1002 and empirically measured properties of the coating composition anda corresponding coating surface. The model M3 would then be re-trainedon the extended training data set.

As an example, the empirical measurement of the composition representedby data point 1002 may show that its predicted coating surface qualityis in the unacceptable coating quality area 1020. The consequence ofre-training on the extended training data set would therefore be thatthe predictive model of the neural network, here graphically visualizedby the dividing line 1016, would adapt in such a way that for acomposition represented by data point 1002, the prediction in the futurewould be that its coating surface quality lies in the area 1020. Thus,by re-training on the extended training data set, the line/plane 1016would be modified in such a way that the line or plane receives a“bulge” such so that the improved model would now recognize and predictthat the composition represented by point 1002 is in the unacceptablecoating quality range 1020. In practice, when selecting the data pointor the corresponding experimental composition, the distance of thecorresponding data points to the separation lines of several propertiesis preferably taken into account, e.g. by selecting the data point withthe minimum average distance to all separation lines/separation planesof the complete multidimensional data space.

The improvement of a model M2 to be used for predicting a coatingcomposition quality based on a complete or incomplete specification ofthe coating composition and/or associated process parameters can beperformed analogously with the only difference that the informationrespectively used as input or output of the model is interchanged.

In a further aspect, disclosed herein is a computer-implemented methodand corresponding system for qualitative and/or quantitativecharacterization of a coating surface. The method and system comprisesteps and features as described in the clauses below.

-   -   1. Clauses:A method for qualitative and/or quantitative        characterization of a coating surface, the method comprising:        -   processing (102) a digital image (604, 606, 1202) of the            coating surface (162-168) by a defect-identification program            (124), the defect-identification program being configured to            recognize patterns, each pattern representing a type of            coating surface defect (1204, 1206, 1306, 1308); and        -   outputting (104) a characterization of the coating surface            by the defect-identification program, the characterization            being computed as a function of coating surface defects            recognized by the defect-identification program during the            processing.    -   2. The method of clause 1, comprising:        -   calculating (106), by the defect-identification program, a            measure (632, 634, 1402) for the recognized defects;            -   wherein the characterization of the coating surface is                computed as a function of the qualitative and/or                quantitative characterization of the measure.    -   3. The method of clause 2,        -   the measure being a quantitative measure selected from a            group comprising: the area of the defect, the number of            bubbles or depressions observed in the digital image (604,            606, 1202), the maximum, minimum and/or average size of the            bubbles or depressions in the digital image (604, 606,            1202); and/or        -   the measure being a qualitative measure, the qualitative            measure being in particular the type of the defect selected            from a group comprising a cratering defect, an abrasion            defect, an adhesion failure defect, an alligatoring defect,            a bleeding defect, a blistering defect, a bloom defect, a            bridging defect, a bubbling defect, a cathodic disbanding            defect, a checking defect, a cissing defect, a cobwebbing            defect, a cracking defect, a crazing defect, a crowsfooting            defect, a delamination defect, a fading defect, a flaking            defect, a grinning defect, a heat defect, an impact defect,            an intercoat contamination defect, a mud cracking defect, an            orange peeling defect, a peeling defect, a pinholes defect,            a rippled coating defect, a runs defect, a rust rashing            defect, a rust spotting defect, a rust staining defect, a            sags defect, a settlement defect, a skinning defect, a            solvent lifting defect, a solvent popping defect, a stress            cracking defect, an undercutting defect, a wrinkling defect.    -   4. The method of any one of the previous clauses, further        comprising:        -   determining at least one coating surface defect type to be            identified;        -   automatically determining one or more illumination angles            and/or one or more image acquisition angles allowing the            acquisition of a digital image that enables the defect            identification program to compute the characterization of            the coating surface depicted in the image in respect to the            at least one determined defect type;        -   positioning one or more light sources (160) at the            determined one or more illumination angles (1502) relative            to the coating surface; and/or        -   positioning one or more cameras, preferably one camera            (134), at the determined one or more image acquisition            angles (1504) relative to the coating surface; and        -   after the positioning of the light source(s), the camera(s)            and/or the coating surface relative to each other, using the            camera(s) for acquiring the digital image of the coating            surface.    -   5. The method of any one of the previous clauses, the processing        of the digital image further comprising:        -   performing, by the defect-identification program, a            classification of the digital image in respect to the type            and/or amount of surface defects depicted therein and/or a            semantic segmentation of the image based on one or more            surface defect types depicted therein and/or an object            detection of defect instances in the image and/or an            instance segmentation of the image, thereby automatically            assigning one or more labels to the whole digital image, to            image regions and/or to individual pixels, each label being            indicative of the type of defect identified in the digital            image; and        -   outputting the one or more assigned labels.    -   6. The method of any one of the previous clauses,        -   the defects-identification program being operatively coupled            to a camera and being configured for            -   determining whether the camera is positioned within a                predefined distance range and/or within a predefined                image acquisition angle range relative to the coating                surface adapted to enable acquisition of images from a                similar relative position as used for acquiring training                images for generating the predictive model of the                defects-identification program;            -   in dependence on the result of the determination,                -   generating a feedback signal for the user and/or the                    camera whether adjustment of the camera position is                    required; and/or                -   automatically adjusting the relative position of the                    camera and the coating surface; and/or                -   enabling the camera to acquire images selectively in                    case the camera is within the predefined distance                    and/or image acquisition angle range.    -   7. The method of any one of the previous clauses, the        defects-identification program being selected from a group        comprising:        -   an app installed on a portable data processing system (130),            in particular a portable telecommunication device, e.g. a            smartphone;        -   an application program installed on a portable or stationary            device (150) specially designed for quality control of            coating surfaces;        -   an application program installed on a high-throughput            facility (244) for the automated or semi-automated            manufacturing and/or testing of coatings;        -   a web application downloaded and instantiated via a network;        -   a program executed within a browser, e.g. a JavaScript            program;        -   a server program instantiated on a server computer, the            server program being operatively coupled via a network            connection to a client program instantiated on a client data            processing system, the client program in particular being            configured for acquiring the digital images and providing            the images via the network to the server program and/or for            displaying the results provided by the server program.    -   8. The method of any one of the previous clauses, the        defect-identification program comprising a predictive model (M1)        having learned from training data (602) in a training step        performed by a machine learning program to recognize the        predefined patterns, the machine learning program being in        particular a neural network.    -   9. The method of clause 8, the machine learning program being a        neural network or a set of neural networks comprising a region        proposal network, the region proposal network being configured        to scan over anchors of an input image for making a proposal        whether the anchor likely contains one of the defect patterns,        the anchors being sub-regions of the input image having anchor        sizes matching expected sizes of the defect patterns.    -   10. The method of any one of clauses 8-9, comprising:        -   performing the training step on the training data (602), the            training data comprising a set of labeled digital training            images (604, 606) of coating surfaces, the labels (616, 618)            identifying the location/positions and/or type of defects in            the training images, the predictive model being trained for            recognizing the pattern by means of the labeled training            images using back propagation.    -   11. The method of clause 10, wherein each of the training images        has assigned additional data (608, 610) being processed in the        training step for enabling the predictive model (M1) to        correlate the additional data with the defect patterns, the        additional data comprising:        -   a quantitative measure of one or more defects depicted in            the training image, e.g., the size and/or severity of the            defect or the number of bubbles;        -   optionally also parameters selected from a group comprising:            -   an indication of one or more components of the coating                used for generating the coating surface depicting in the                training image;            -   an indication of an absolute or relative amount of one                or more of the components of the coating composition;                and/or            -   one or more manufacturing-process parameters, the                manufacturing-process parameters characterizing a                process of generating a coating composition, the process                parameters for example comprising mixing speed and/or                mixing duration of the coating composition; and/or            -   one or more application-process parameters, the                application-process parameters characterizing a process                of applying a coating composition on a substrate, the                application-process parameters in particular comprising                the amount of coating composition applied per area of                the coating surface, the type of substrate and/or or the                type of application devices; and/or            -   system parameters of an imaging system used for                acquiring the training images, the system parameters                being selected from a group comprising type of light                source(s) used for illuminating the coating surface,                brightness of the light source(s), illumination angle,                wavelength of the light source(s), type of one or more                cameras used for acquiring the digital image of the                coating surface, image acquisition angle(s), position(s)                of the one or more camera(s).

In a further aspect, disclosed herein is a computer-implemented methodand corresponding system for providing a coating composition-relatedprediction program. The method and system comprise steps and features asdescribed in the clauses below.

Clauses:

-   -   12. A computer-implemented method for providing a coating        composition-related prediction program, the method comprising:        -   providing a database (204, 904) comprising associations of            qualitative and/or quantitative characterizations of coating            surfaces in association with one or more parameters selected            from the group comprising one or more of the components of            the coating composition used for producing the respective            coating surface, relative and/or absolute amounts of one or            more of the said components, manufacturing-process            parameters of the coating composition and/or            application-process parameters used for creating the coating            surfaces;        -   training a machine learning model on the associations of the            coating surface characterizations with the one or more            parameters in the database for providing a predictive model            (M2, M3) having learned to correlate qualitative and/or            quantitative characterizations of one or more coating            surfaces with one or more of the parameters stored in            association with the respective coating surface            characterizations; and        -   providing a composition-quality-prediction program which            comprises the predictive model (M2), the            composition-quality-prediction program being configured for            using the predictive model (M2) for predicting the            properties of a coating surface to be produced from one or            more input parameters selected from the group comprising one            or more components of a coating composition to be used for            producing a coating surface, relative and/or absolute            amounts of one or more of the said components,            manufacturing-process parameters to be used for preparing            the coating composition and/or application-process            parameters to be used for creating the coating surface;            and/or        -   providing a composition-specification-prediction program            which comprises the predictive model (M3), the            composition-specification-prediction program being            configured for using the predictive model (M3) for            predicting, based on an input specifying at least a desired            coating surface characterization, one or more output            parameters related to a coating composition predicted to            generate a coating surface having the input surface            characterizations, the one or more output parameters being            selected from the group comprising: one or more components            of the said coating composition, relative and/or absolute            amounts of one or more of the said components,            manufacturing-process parameters to be used for preparing            the coating composition and/or application-process            parameters for creating the coating surface, wherein            optionally the composition-specification-prediction program            is configured for receiving an incomplete coating            composition specification and for using the specification            for limiting the solution space of the predicted output            parameters.    -   13. The method of clause 12, the method comprising:        -   providing a plurality of images depicting coating surfaces            made from multiple different coating compositions, wherein            at least some of the coating surfaces respectively have one            or more coating defects of one or more different defect            types;        -   applying a defect-identification program on the images for            recognizing patterns in the images, for obtaining the            measures of the coating defects represented by the            identified patterns in the images and for computing a            qualitative and/or quantitative characterization of the            coating surfaces depicted in the images, wherein the            defect-identification program preferably is the            defect-identification program specified in any one of            clauses 1-11;        -   storing the qualitative and/or quantitative            characterizations of the coating surfaces in association            with one or more parameters related to the coating            composition used for creating the coating surface comprising            these defects in the database.    -   14. The method of any one of the previous clauses 12-13, further        comprising using of the composition-specification-prediction        program for predicting a coating composition specification        meeting a desired coating surface characterization provided as        input, the using comprising:        -   providing at least a specification of a desired coating            surface characterization as input to the            composition-specification-prediction program;        -   predicting, by the composition-specification-prediction            program, a specification of a coating composition adapted to            provide a coating surface having the desired surface            characteristics, wherein the specification comprises one or            more parameters selected from a group comprising: one or            more coating composition components, absolute or relative            amounts of one or more of the coating composition            components, manufacturing process parameters and/or            application process parameters;        -   preferably further comprising outputting the predicted            specification of the coating composition to a human and/or            inputting the specification of the selected candidate            coating composition to a processor which controls a facility            (244) for producing and/or testing compositions for coating            compositions, wherein the processor drives the facility to            produce the input coating composition.    -   15. The method of any one of the previous clauses 12-14, wherein        the training process involves an active learning module, the        method further comprising:        -   a. performing the training of the machine learning model,            preferably a neural network, on the associations of the            measures and the coating components in the database for            providing the predictive model (M2, M3), where a loss            function is minimized for the training,        -   b. testing to determine whether the value of the loss            function obtained for the predictive model meets a specified            criterion, whereby selectively in the event that (meaning            only in case of that) the criterion is not met, the            following steps i-v are carried out:            -   i. selection of a candidate coating composition                specification (1002) from a plurality of candidate                coating composition specifications by the active                learning module, the selected candidate composition                specification specifying the one of the candidate                compositions determined to provide the highest learning                effect of the predictive model (M2, M3) regarding the                correlation of qualitative and/or quantitative coating                surface characterizations and one or more parameters                selected from a group comprising coating components,                component amounts, manufacturing process parameters                and/or application process parameters of the coating                composition;            -   ii. driving the facility by the computer system for                automatically producing the candidate coating                composition in accordance with the selected                specification, for automatically applying the produced                candidate composition on a substrate and for                automatically acquiring an image of the surface with the                applied coating composition;            -   iii. applying the defects-identification program on the                image acquired in step ii for computing and storing a                qualitative or quantitative characterization of the                coating surface depicted in said image in the database,                thereby extending the database;            -   iv. re-training of the predictive model (M2, M3) on the                extended database for providing an improved version of                the predictive model,            -   v. repeated execution of step b using the improved                version of the predictive model,        -   c. replacing the predictive model (M2) of the            composition-quality-prediction program with the improved            version of the predictive model and/or replacing the            predictive model (M3) of the            composition-specification-prediction program with the            improved version of the predictive model.    -   16. The method of any one of the previous clauses 12-14, wherein        the training process involves an active learning module, the        method further comprising        -   using the active learning module to interactively query the            predictive model M2) of the composition-quality-prediction            program to perform multiple predictions of the quality of a            surface coating to be generated by a coating composition,            each prediction being based on a specification of a            candidate composition comprising one or more components of a            coating composition, relative and/or absolute amounts of one            or more of the said components, and/or manufacturing-process            parameters to be used for preparing the candidate coating            composition and/or application-process parameters to be used            for creating the candidate coating surface;        -   determining, by the active learning module, a degree of            uncertainty of each of the prediction (e.g. by means of a            minimum marginal hyperplane method, a loss function, etc.);        -   identifying the one or more candidate composition            specifications which yielded a prediction result whose            degree of uncertainty exceeds a predefined uncertainty            threshold;        -   producing candidate coating compositions selectively in            accordance with the identified candidate coating            specifications (but not the other candidate coating            composition whose prediction had a sufficient degree of            certainty, i.e., an uncertainty level below the threshold),            thereby automatically generating new coating compositions,        -   applying the new coating compositions to obtain new coating            surfaces,        -   acquire images of the new coating surfaces;        -   performing an image analysis of the acquired images of            identifying and/or characterizing surface defects of the new            coating surfaces; and        -   re-training the composition-quality-prediction program,            thereby taking into account also the identified candidate            coating specifications and their respectively obtained            surface coating defect characteristics for improving the            accuracy of the composition-quality-prediction program.

For example, the generation of the new coating compositions andoptionally also the generation of the coating surfaces, and optionallyalso the image acquisition and analysis can be obtained by driving thefacility (244) by the computer system to cause the facility toautomatically produce the candidate coating compositions accordance withthe identified candidate coating specifications (but not the othercandidate coating composition whose prediction had a sufficient degreeof certainty, i.e., an uncertainty level below the threshold). Thefacility automatically applies the produced candidate coatingcomposition(s) on a respective substrate. The facility may comprise orbe operatively coupled to an image acquisition system. The imageacquisition system is used for acquiring an image of the surface withthe applied coating composition. The image acquisition system may applythe defects-identification program on the acquired images of the surfacecoatings of the generated and applied identified candidate coatingcomposition as described herein for embodiments of the invention.Thereby, a qualitative or quantitative characteristics of the newcandidate coating surfaces depicted in each image is obtained. Thesecharacteristics can be used for re-training and improving the predictivemodel (M2). A prediction result with a very high degree of uncertaintyis interpreted by the active learning module as an indication that thecandidate coating composition specification represents a coatingcomposition which is highly unfamiliar to the predictive model, so thepredictive model could achieve a high learning effect based on real,empirical and hence reliable data obtained for such a candidate coatingcomposition. The predictive model (M3) of thecomposition-specification-prediction program can be improved using theactive learning module analogously. The only difference is that theinput of the predictive model (M3) of thecomposition-specification-prediction program corresponds to the outputof the predictive model (M2) of the composition-quality-predictionprogram and vice versa, so the two models are challenged based ondifferent candidate input data. In the case of the model (M3), thecandidate input data is a specification of desired physical or chemicalproperty of the coating composition, in particular a qualitative and/orquantitative characteristic of a surface coating generated from thiscoating composition. The other steps and aspects are identical.

-   -   17. A system comprising        -   a facility (244) for producing and testing compositions for            paints, varnishes, printing inks, grinding resins, pigment            concentrates or other coating materials, where the facility            comprises at least two workstations, where the at least two            workstations are connected to one another via a transport            system on which self-propelled transport vehicles are able            to run for transporting the components of the composition            and/or of the composition produced between the workstations,            and        -   a computer system (224) configured to perform the method of            any one of clauses 1-16.

The features of the methods and systems described herein for theabove-mentioned clauses can freely be combined with embodiments andexamples of the invention as long as they are not mutually exclusive.

LIST OF REFERENCE NUMERALS

102-118 steps

120 data processing system

122 data storage medium

124 defect-identification program

125 digital image

126 processor(s)

130 portable telecommunication device

132 GUI of defect-identification program

134 camera

136 GUI of defect-identification program

140 browser

142 network

144 server computer

146 webserver

150 coating quality control device

152 control panel

154 carrier/transportation belt

156 robotic arm

158 robotic arm

160 light source

162-168 coated samples

170 computer system

204 database

244 facility for manufacturing and/or testing coating compositions

246 main control computer

248 control unit

252 image acquisition unit

254 sample coating unit

246 mixing units

257 analyzers

258 transport unit

400 structure of neural network

402 input data (e.g. vector)

404 layers of neural network

406 output of predictive model

602 training data for predictive model M1

604 Training image for predictive model M1

606 Training image for predictive model M1

608 label with context parameters

610 label with context parameters

612 predictive model M1

614 GUI of defect-identification program

615 camera control module

616 manual label with defects measures and quality characterizations

618 manual label with defects measures and quality characterizations

620 database or image acquisition system

622 test data for predictive model M1

624 test image for predictive model M1

626 test image for predictive model M1

628 label with context parameters

630 label with context parameters

632 computed label with defects measures and quality characterizations

634 computed label with defects measures and quality characterizations

711 untrained predictive model M2

712 trained predictive model M2

714 composition-quality-prediction-program

715 GUI of composition-quality-prediction-program

720 database

721 active learning module

722 test data for M2

728 context parameters

729 computed rheological properties

730 context parameters

731 computed rheological properties

732 computed coating quality/properties

734 computed coating quality/properties

736 output of composition-quality-prediction program

802-816 steps

900 system

902 DBMS

904 database

906 known compositions

908 candidate compositions

910 known composition and properties

912 selected candidate composition

914 selection

916 selected composition

918 properties

944 facility

946 control computer

948 control software

952 analysis unit

954 preparation unit

956 preparation unit

958 transport unit

1000 plot of hyperplane

1002-1009 data points

1016 separation line

1018 area of acceptable coating quality

1020 area of inacceptable coating quality

1202 digital image of coating surface

1204 bubble

1206 bubble

1208 dirt

1210 manually labelled image 1202

1212 manually added labels

1302 digital image of coating surface

1306 foam cavity

1308 foam cavity

1304 automatically generated labels/segment borders of computationallyidentified cavity defects

1402 quantitative measure of a bubble defect: histogram

1502 illumination angles

1504 image capturing angles

1. A computer-implemented method for providing a coatingcomposition-related prediction program, the method comprising: providinga database comprising associations of one or more of qualitative andquantitative characterizations of coating surfaces and one or moreparameters selected from a group comprising one or more of: one or moreof components of coating composition used for producing a respectivecoating surface; one or more of relative and absolute amounts of one ormore of the components; manufacturing-process parameters of the coatingcomposition, and application-process parameters used for creating thecoating surfaces; training a machine learning model on the associationsof the coating surface characterizations with the one or more parametersin the database for providing a predictive model having learned tocorrelate one or more of the qualitative and the quantitativecharacterizations of one or more coating surfaces with one or more ofthe parameters stored in association with respective coating surfacecharacterizations; and one or more of: providing acomposition-quality-prediction program which comprises a firstpredictive model, the composition-quality-prediction program beingconfigured for using the first predictive model for predicting theproperties of a coating surface to be produced from one or more inputparameters selected from a group comprising one or more of: one or morecomponents of the coating composition to be used for producing thecoating surface; one or more of relative and absolute amounts of one ormore of the components; manufacturing-process parameters to be used forpreparing the coating composition; application-process parameters to beused for creating the coating surface; and providing acomposition-specification-prediction program which comprises a secondpredictive model, the composition-specification-prediction program beingconfigured for using the second predictive model for predicting, basedon an input specifying at least a desired coating surfacecharacterization, one or more output parameters related to the coatingcomposition predicted to generate the coating surface having inputsurface characterizations, the one or more output parameters beingselected from a group comprising one or more of: one or more componentsof the coating composition; one or more of relative and absolute amountsof one or more of the components; manufacturing-process parameters to beused for preparing the coating composition; and application-processparameters for creating the coating surface, wherein optionally thecomposition-specification-prediction program is configured for receivingan incomplete coating composition specification and for using theincomplete coating composition specification for limiting a solutionspace of predicted output parameters.
 2. The method of claim 1, themethod comprising: providing a images depicting coating surfaces madefrom multiple different coating compositions, wherein at least some ofthe coating surfaces respectively have one or more coating defects ofone or more different defect types; applying a defect-identificationprogram on the images for recognizing patterns in the images, forobtaining the measures of the coating defects represented by theidentified patterns in the images and for computing one or more of thequalitative and a the quantitative characterization of the coatingsurfaces depicted in the images; storing one or more of the qualitativeand the quantitative characterizations of the coating surfaces inassociation with one or more parameters related to the coatingcomposition used for creating the coating surface comprising thesedefects in the database.
 3. The method of claim 1, wherein the coatingcomposition comprises a liquid basic component and one or more solidsdispersed in the liquid basic component, wherein the liquid basiccomponent comprises one or more of water, a water-based solution or anorganic solvent, wherein the one or more solids are selected from agroup comprising pigments, dyes, and fillers.
 4. The method of claim 1,wherein the properties of the coating surface comprise an indication ofone or more coating defects selected from a group comprising one or moreof: a cratering defect, an abrasion defect, an adhesion failure defect,an alligatoring defect, a bleeding defect, a blistering defect, a bloomdefect, a bridging defect, a bubbling defect, a cathodic disbandingdefect, a checking defect, a cissing defect, a cobwebbing defect, acracking defect, a crazing defect, a crowsfooting defect, a delaminationdefect, a fading defect, a flaking defect, a grinning defect, a heatdefect, an impact defect, an intercoat contamination defect, a mudcracking defect, an orange peeling defect, a peeling defect, a pinholesdefect, a rippled coating defect, a runs defect, a rust rashing defect,a rust spotting defect, a rust staining defect, a sags defect, asettlement defect, a skinning defect, a solvent lifting defect, asolvent popping defect, a stress cracking defect, an undercuttingdefect, and a wrinkling defect.
 5. The method of claim 1, furthercomprising using of the composition-specification-prediction program forpredicting a coating composition specification meeting a desired coatingsurface characterization provided as input, the using comprising:providing at least a specification of a desired coating surfacecharacterization as input to the composition-specification-predictionprogram; predicting, by the composition-specification-predictionprogram, a specification of a coating composition adapted to provide acoating surface having the desired surface characteristics, wherein thespecification comprises one or more parameters selected from a groupcomprising one or more of: one or more coating composition components;absolute or relative amounts of one or more of the coating compositioncomponents; manufacturing process parameters: and application processparameters; preferably further comprising one or more of: outputting thepredicted specification of the coating composition to a human; andinputting the specification of the selected candidate coatingcomposition to a processor which controls a facility for one or more ofproducing and testing compositions for coating compositions, wherein theprocessor drives the facility to produce the input coating composition.6. The method of claim 1, wherein the training the machine learningmodel comprises using an active learning module for selecting acandidate coating composition specification from a set of candidatecomposition specifications which provides the strongest learning effectfor the first predictive model and the second predictive model, thelearning effect being provided as a result of the preparation of theselected candidate composition, empirical measurement of properties ofthe selected candidate composition, and retraining of one or more of thefirst predictive model and the second predictive model on training datacomprising the empirically measured properties of the selected andprepared candidate composition.
 7. The method of claim 1, wherein thetraining the machine learning model involves an active learning module,the method further comprising: a. performing the training of the machinelearning model on the associations of the measures and the coatingcomponents in the database for providing the first predictive model andthe second predictive model, where a loss function is minimized for thetraining, b. testing to determine whether a value of the loss functionobtained for the predictive model meets a specified criterion, wherebyselectively, in the event that the specified criterion is not met, thefollowing steps are carried out: i. selection of a candidate coatingcomposition specification from a plurality of candidate coatingcomposition specifications by the active learning module, the selectedcandidate composition specification specifying the one of the candidatecompositions determined to provide the highest learning effect of thefirst predictive model and the second predictive model regarding thecorrelation of one or more of the qualitative and the quantitativecoating surface characterizations and one or more parameters selectedfrom a group comprising one or more of: coating components; componentamounts; manufacturing process parameters, and application processparameters of the coating composition; ii. driving a facility forautomatically producing the candidate coating composition in accordancewith the selected specification, for automatically applying the producedcandidate composition on a substrate and for automatically acquiring animage of the surface with the applied coating composition; iii. applyingthe defects-identification program on the image acquired in step ii forcomputing and storing one or more of the qualitative and thequantitative characterization of the coating surface depicted in theimage in the database, thereby extending the database; iv. re-trainingof the first predictive model and the second predictive model on theextended database for providing an improved version of the predictivemodel, v. repeated execution of step b using the improved version of thepredictive model, c. one or more of: replacing the first predictivemodel of the composition-quality-prediction program with the improvedversion of the first predictive model, and replacing the secondpredictive model of the composition-specification-prediction programwith the improved version of the second predictive model.
 8. The methodof claim 6, wherein the selection of the candidate coating compositionspecification by the active learning module comprises for each of themultiple candidate coating composition specifications: predictingproperties of the specified candidate coating composition using thefirst predictive model of the composition-quality-prediction program;determining an uncertainty score of the prediction; and using the one ofthe multiple candidate coating compositions as the selected candidatecoating composition specification having the maximum uncertainty score.9. The method of claim 6, wherein the candidate coating compositionspecification in addition comprises at least a desired coating surfacecharacterization, wherein the selection of the candidate coatingcomposition specification by the active learning module comprises foreach of the multiple candidate coating composition specifications:Predicting one or more of the output parameters predicted to be relatedto the candidate coating composition having the desired coating surfacecharacterization using the second predictive model of thecomposition-specification-prediction program; Determining an uncertaintyscore of the prediction; and using the one of the multiple candidatecoating compositions as the selected candidate coating compositionspecification having the maximum uncertainty score.
 10. The method ofclaim 6, wherein the active learning module is operatively coupled to afacility for preparing and testing compositions for paints, varnishes,printing inks, grinding resins, pigment concentrates or other coatingmaterials.
 11. The method of claim 10, wherein the active learningmodule is configured to automatically drive the facility iteratively togenerate and empirically analyze one or more candidate coatingcompositions until the accuracy of the first predictive model and thesecond predictive model reaches a predefined minimum accuracy thresholdor until a maximum number of iterations is reached.
 12. The method ofclaim 10, wherein the active learning module is configured to generate agraphical user interface (GUI), the GUI comprising a plurality ofselectable GUI elements, each GUI element representing one of thecandidate coating composition specifications, the GUI comprising a modelquality indicator being indicative of the accuracy of the firstpredictive model and the second predictive model, the active learningmodule being configured for driving the facility, upon a selection ofone of the GUI-elements by a user, to prepare and test the candidatecoating composition represented by the selected GUI element, use theempirical properties of the prepared candidate coating composition forretraining the predictive model; and update the model quality indicatorof the GUI to indicate the accuracy of the retrained predictive model.13. A computer system comprising at least one processor andcomputer-interpretable instructions which, when executed by the at leastone processor, cause the processor to: access a database comprisingassociations of one or more of qualitative and quantitativecharacterizations of coating surfaces and one or more parametersselected from a group comprising one or more of: one or more of thecomponents of the coating composition used for producing the respectivecoating surface; one or more of relative and absolute amounts of one ormore of the components; manufacturing-process parameters of the coatingcomposition; and application-process parameters used for creating thecoating surfaces; train a machine learning model on the associations ofthe coating surface characterizations with the one or more parameters inthe database for providing a predictive model having learned tocorrelate the one or more of the qualitative and the quantitativecharacterizations of one or more coating surfaces with one or more ofthe parameters stored in association with the respective coating surfacecharacterizations; and one or more of: provide acomposition-quality-prediction program which comprises a firstpredictive model, the composition-quality-prediction program beingconfigured for using the first predictive model for predicting theproperties of a coating surface to be produced from one or more inputparameters selected from a group comprising one or more of: one or morecomponents of a coating composition to be used for producing a coatingsurface; one or more of relative and absolute amounts of one or more ofthe components; manufacturing-process parameters to be used forpreparing the coating composition; and application-process parameters tobe used for creating the coating surface; and provide acomposition-specification-prediction program which comprises a secondpredictive model, the composition-specification-prediction program beingconfigured for using the second predictive model for predicting, basedon an input specifying at least a desired coating surfacecharacterization and outputting one or more parameters related to acoating composition predicted to generate a coating surface having theinput surface characterizations, the one or more output parameters beingselected from a group comprising one or more of: one or more componentsof the coating composition: one or more of relative and absolute amountsof one or more of the components; manufacturing-process parameters to beused for preparing the coating composition; and application-processparameters for creating the coating surface, wherein optionally thecomposition-specification-prediction program is configured for receivingan incomplete coating composition specification and for using theincomplete coating composition specification for limiting a solutionspace of a predicted output parameters.
 14. A system comprising afacility for producing and testing compositions for paints, varnishes,printing inks, grinding resins, pigment concentrates or other coatingmaterials, where the facility comprises at least two workstations, wherethe at least two workstations are connected to one another via atransport system on which self-propelled transport vehicles are able torun for transporting one or more of the components of the compositionand of the composition produced between the workstations, and thecomputer system of claim
 13. 15. (canceled)
 16. A non-transitorymachine-readable data structure encoding at least one coatingcomposition specification, the data structure being an output of anactive learning module and being interpretable by a facility forproducing and testing coating compositions, the data structure beingconfigured to cause, upon being input to the facility, the facility toprepare and test the coating composition specified in the coatingcomposition specification.